Merge pull request #601 from trapexit/hidden

remove .fuse_hidden file creation

README.md

@@ -1,6 +1,6 @@
 % mergerfs(1) mergerfs user manual
 % Antonio SJ Musumeci <trapexit@spawn.link>
-% 2019-02-22
+% 2019-03-21
 
 # NAME
 
@@ -68,7 +68,6 @@ mergerfs does **not** support the copy-on-write (CoW) behavior found in **aufs**
 * **minfreespace=value**: the minimum space value used for creation policies. Understands 'K', 'M', and 'G' to represent kilobyte, megabyte, and gigabyte respectively. (default: 4G)
 * **moveonenospc=true|false**: when enabled (set to **true**) if a **write** fails with **ENOSPC** or **EDQUOT** a scan of all drives will be done looking for the drive with the most free space which is at least the size of the file plus the amount which failed to write. An attempt to move the file to that drive will occur (keeping all metadata possible) and if successful the original is unlinked and the write retried. (default: false)
 * **use_ino**: causes mergerfs to supply file/directory inodes rather than libfuse. While not a default it is recommended it be enabled so that linked files share the same inode value.
-* **hard_remove**: force libfuse to immedately remove files when unlinked. This will keep the `.fuse_hidden` files from showing up but if software uses an opened but unlinked file in certain ways it could result in errors.
 * **dropcacheonclose=true|false**: when a file is requested to be closed call `posix_fadvise` on it first to instruct the kernel that we no longer need the data and it can drop its cache. Recommended when **direct_io** is not enabled to limit double caching. (default: false)
 * **symlinkify=true|false**: when enabled (set to **true**) and a file is not writable and its mtime or ctime is older than **symlinkify_timeout** files will be reported as symlinks to the original files. Please read more below before using. (default: false)
 * **symlinkify_timeout=value**: time to wait, in seconds, to activate the **symlinkify** behavior. (default: 3600)
@@ -172,7 +171,7 @@ Runtime extended attribute support can be managed via the `xattr` option. By def
 
 The POSIX filesystem API is made up of a number of functions. **creat**, **stat**, **chown**, etc. In mergerfs most of the core functions are grouped into 3 categories: **action**, **create**, and **search**. These functions and categories can be assigned a policy which dictates what file or directory is chosen when performing that behavior. Any policy can be assigned to a function or category though some may not be very useful in practice. For instance: **rand** (random) may be useful for file creation (create) but could lead to very odd behavior if used for `chmod` if there were more than one copy of the file.
 
-Some functions, listed in the category `N/A` below, can not be assigned the normal policies. All functions which work on file handles use the handle which was acquired by `open` or `create`. `readdir` has no real need for a policy given the purpose is merely to return a list of entries in a directory. `statfs`'s behavior can be modified via other options.
+Some functions, listed in the category `N/A` below, can not be assigned the normal policies. All functions which work on file handles use the handle which was acquired by `open` or `create`. `readdir` has no real need for a policy given the purpose is merely to return a list of entries in a directory. `statfs`'s behavior can be modified via other options. That said many times the current FUSE kernel driver will not always provide the file handle when a client calls `fgetattr`, `fchown`, `fchmod`, `futimens`, `ftruncate`, etc. This means it will call the regular, path based, versions.
 
 When using policies which are based on a branch's available space the base path provided is used. Not the full path to the file in question. Meaning that sub mounts won't be considered in the space calculations. The reason is that it doesn't really work for non-path preserving policies and can lead to non-obvious behaviors.
 
@@ -182,8 +181,8 @@ When using policies which are based on a branch's available space the base path
 |----------|-------------------------------------------------------------------------------------|
 | action   | chmod, chown, link, removexattr, rename, rmdir, setxattr, truncate, unlink, utimens |
 | create   | create, mkdir, mknod, symlink                                                       |
-| search   | access, getattr, getxattr, ioctl, listxattr, open, readlink                         |
+| search   | access, getattr, getxattr, ioctl (directories), listxattr, open, readlink                         |
-| N/A      | fallocate, fgetattr, fsync, ftruncate, ioctl, read, readdir, release, statfs, write |
+| N/A      | fchmod, fchown, futimens, ftruncate, fallocate, fgetattr, fsync, ioctl (files), read, readdir, release, statfs, write |
 
 In cases where something may be searched (to confirm a directory exists across all source mounts) **getattr** will be used.
 
@@ -239,6 +238,24 @@ If all branches are filtered an error will be returned. Typically **EROFS** or *
 | search   | ff     |
 
 
+#### ioctl
+
+When `ioctl` is used with an open file then it will use the file handle which was created at the original `open` call. However, when using `ioctl` with a directory mergerfs will use the `open` policy to find the directory to act on.
+
+
+#### unlink
+
+In FUSE there is an opaque "file handle" which is created by `open`, `create`, or `opendir`, passed to the kernel, and then is passed back to the FUSE userland application by the kernel. Unfortunately, the FUSE kernel driver does not always send the file handle when it theoretically could/should. This complicates certain behaviors / workflows particularly in the high level API. As a result mergerfs is currently doing a few hacky things.
+
+libfuse2 and libfuse3, when using the high level API, will rename names to `.fuse_hiddenXXXXXX` if the file is open when unlinked or renamed over. It does this so the file is still available when a request referencing the now missing file is made. This file however keeps a `rmdir` from succeeding and can be picked up by software reading directories.
+
+The change mergerfs has done is that if a file is open when an unlink or rename happens it will open the file and keep it open till closed by all those who opened it prior. When a request comes in referencing that file and it doesn't include a file handle it will instead use the file handle created at unlink/rename time.
+
+This won't result in technically proper behavior but close enough for many usecases.
+
+The plan is to rewrite mergerfs to use the low level API so these invasive libfuse changes are no longer necessary.
+
+
 #### rename & link ####
 
 **NOTE:** If you're receiving errors from software when files are moved / renamed / linked then you should consider changing the create policy to one which is **not** path preserving, enabling `ignorepponrename`, or contacting the author of the offending software and requesting that `EXDEV` be properly handled.
@@ -765,11 +782,11 @@ There is a bug in the kernel. A work around appears to be turning off `splice`.
 
 #### rm: fts_read failed: No such file or directory
 
+NOTE: This is only relevant to mergerfs versions at or below v2.25.x and should not occur in more recent versions. See the notes on `unlink`.
+
 Not *really* a bug. The FUSE library will move files when asked to delete them as a way to deal with certain edge cases and then later delete that file when its clear the file is no longer needed. This however can lead to two issues. One is that these hidden files are noticed by `rm -rf` or `find` when scanning directories and they may try to remove them and they might have disappeared already. There is nothing *wrong* about this happening but it can be annoying. The second issue is that a directory might not be able to removed on account of the hidden file being still there.
 
-Using the **hard_remove** option will make it so these temporary files are not used and files are deleted immedately. That has a side effect however. Files which are unlinked and then they are still used (in certain forms) will result in an error.
+Using the **hard_remove** option will make it so these temporary files are not used and files are deleted immedately. That has a side effect however. Files which are unlinked and then they are still used (in certain forms) will result in an error (ENOENT).
-
-A fix is in the works for this.
 
 
 # FAQ
@@ -821,7 +838,15 @@ This can be especially apparent when filling an empty pool from an external sour
 
 #### Why was libfuse embedded into mergerfs?
 
-A significant number of users use mergerfs on distros with old versions of libfuse which have serious bugs. Requiring updated versions of libfuse on those distros isn't pratical (no package offered, user inexperience, etc.). The only practical way to provide a stable runtime on those systems was to "vendor" the library into the project.
+1. A significant number of users use mergerfs on distros with old versions of libfuse which have serious bugs. Requiring updated versions of libfuse on those distros isn't pratical (no package offered, user inexperience, etc.). The only practical way to provide a stable runtime on those systems was to "vendor" / embed the library into the project.
+2. mergerfs was written to use the high level API. There are a number of limitations in the HLAPI that make certain features difficult or impossible to implement. While some of these features could be patched into newer versions of libfuse without breaking the public API some of them would require hacky code to provide backwards compatibility. While it may still be worth working with upstream to address these issues in future versions, since the library needs to be vendored for stability and compatibility reasons it is preferable / easier to modify the API. Longer term the plan is to rewrite mergerfs to use the low level API.
+
+
+#### Why did support for system libfuse get removed?
+
+See above first.
+
+If/when mergerfs is rewritten to use the low-level API then it'll be plausible to support system libfuse but till then its simply too much work to manage the differences across the versions.
 
 
 #### Why use mergerfs over mhddfs?
@@ -920,7 +945,9 @@ You could also set `xattr` to `noattr` or `nosys` to short circuit or stop all x
 
 #### What are these .fuse_hidden files?
 
-When not using `hard_remove` libfuse will create .fuse_hiddenXXXXXXXX files when an opened file is unlinked. This is to simplify "use after unlink" usecases. There is a possibility these files end up being picked up by software scanning directories and not ignoring hidden files. This is rarely a problem but a solution is in the works.
+NOTE: mergerfs >= 2.26.0 will not have these temporary files. See the notes on `unlink`.
+
+When not using **hard_remove** libfuse will create .fuse_hiddenXXXXXXXX files when an opened file is unlinked. This is to simplify "use after unlink" usecases. There is a possibility these files end up being picked up by software scanning directories and not ignoring hidden files. This is rarely a problem but a solution is in the works.
 
 The files are cleaned up once the file is finally closed. Only if mergerfs crashes or is killed would they be left around. They are safe to remove as they are already unlinked files.
 

libfuse/include/fuse.h

@@ -123,8 +123,17 @@ struct fuse_operations {
 	 * */
 	int (*mkdir) (const char *, mode_t);
 
+       /** Hide files unlinked / renamed over
+        *
+        * Allows storing of a file handle when a file is unlinked
+        * while open. Helps manage the fact the kernel usually does
+        * not send fh with getattr requests.
+        */
+        int (*prepare_hide)(const char *name_, uint64_t *fh_, int type_);
+        int (*free_hide)(const uint64_t fh_);
+
 	/** Remove a file */
-	int (*unlink) (const char *);
+        int (*unlink) (const char *);
 
 	/** Remove a directory */
 	int (*rmdir) (const char *);
@@ -140,9 +149,11 @@ struct fuse_operations {
 
 	/** Change the permission bits of a file */
 	int (*chmod) (const char *, mode_t);
+        int (*fchmod)(const struct fuse_file_info *, const mode_t);
 
 	/** Change the owner and group of a file */
 	int (*chown) (const char *, uid_t, gid_t);
+        int (*fchown)(const struct fuse_file_info *, const uid_t, const gid_t);
 
 	/** Change the size of a file */
 	int (*truncate) (const char *, off_t);
@@ -441,7 +452,8 @@ struct fuse_operations {
 	 *
 	 * Introduced in version 2.6
 	 */
-	int (*utimens) (const char *, const struct timespec tv[2]);
+	int (*utimens)(const char *, const struct timespec tv[2]);
+        int (*futimens)(const struct fuse_file_info *ffi_, const struct timespec tv_[2]);
 
 	/**
 	 * Map block index within file to block index within device

libfuse/lib/fuse.c

@@ -64,7 +64,7 @@ struct fuse_config {
 	int remember;
 	int nopath;
 	int debug;
-	int hard_remove;
+        int hard_remove; /* not used */
 	int use_ino;
 	int readdir_ino;
 	int set_mode;
@@ -176,8 +176,9 @@ struct node {
 	struct timespec mtime;
 	off_t size;
 	struct lock *locks;
-	unsigned int is_hidden : 1;
+        uint64_t hidden_fh;
-	unsigned int cache_valid : 1;
+        char is_hidden;
+        char cache_valid;
 	int treelock;
 	char inline_name[32];
 };
@@ -1244,26 +1245,21 @@ static void remove_node(struct fuse *f, fuse_ino_t dir, const char *name)
 }
 
 static int rename_node(struct fuse *f, fuse_ino_t olddir, const char *oldname,
-		       fuse_ino_t newdir, const char *newname, int hide)
+		       fuse_ino_t newdir, const char *newname)
 {
 	struct node *node;
 	struct node *newnode;
 	int err = 0;
 
 	pthread_mutex_lock(&f->lock);
-	node  = lookup_node(f, olddir, oldname);
+	node = lookup_node(f, olddir, oldname);
-	newnode	 = lookup_node(f, newdir, newname);
+	newnode	= lookup_node(f, newdir, newname);
 	if (node == NULL)
 		goto out;
 
-	if (newnode != NULL) {
+	if (newnode != NULL)
-		if (hide) {
-			fprintf(stderr, "fuse: hidden file got created during hiding\n");
-			err = -EBUSY;
-			goto out;
-		}
 		unlink_node(f, newnode);
-	}
+
 
 	unhash_name(f, node);
 	if (hash_name(f, node, newdir, newname) == -1) {
@@ -1271,9 +1267,6 @@ static int rename_node(struct fuse *f, fuse_ino_t olddir, const char *oldname,
 		goto out;
 	}
 
-	if (hide)
-		node->is_hidden = 1;
-
 out:
 	pthread_mutex_unlock(&f->lock);
 	return err;
@@ -1534,6 +1527,32 @@ int fuse_fs_rename(struct fuse_fs *fs, const char *oldpath,
 	}
 }
 
+int
+fuse_fs_prepare_hide(struct fuse_fs *fs_,
+                     const char     *path_,
+                     uint64_t       *fh_,
+                     int             type_)
+{
+  fuse_get_context()->private_data = fs_->user_data;
+
+  if(fs_->op.prepare_hide)
+    return fs_->op.prepare_hide(path_,fh_,type_);
+
+  return -ENOSYS;
+}
+
+int
+fuse_fs_free_hide(struct fuse_fs *fs_,
+                  uint64_t        fh_)
+{
+  fuse_get_context()->private_data = fs_->user_data;
+
+  if(fs_->op.free_hide)
+    return fs_->op.free_hide(fh_);
+
+  return -ENOSYS;
+}
+
 int fuse_fs_unlink(struct fuse_fs *fs, const char *path)
 {
 	fuse_get_context()->private_data = fs->user_data;
@@ -2007,6 +2026,20 @@ int fuse_fs_chown(struct fuse_fs *fs, const char *path, uid_t uid, gid_t gid)
 	}
 }
 
+int
+fuse_fs_fchown(struct fuse_fs              *fs_,
+               const struct fuse_file_info *ffi_,
+               const uid_t                  uid_,
+               const gid_t                  gid_)
+{
+  fuse_get_context()->private_data = fs_->user_data;
+
+  if(fs_->op.fchown)
+    return fs_->op.fchown(ffi_,uid_,gid_);
+
+  return -ENOSYS;
+}
+
 int fuse_fs_truncate(struct fuse_fs *fs, const char *path, off_t size)
 {
 	fuse_get_context()->private_data = fs->user_data;
@@ -2069,6 +2102,19 @@ int fuse_fs_utimens(struct fuse_fs *fs, const char *path,
 	}
 }
 
+int
+fuse_fs_futimens(struct fuse_fs *fs_,
+                 const struct fuse_file_info *ffi_,
+                 const struct timespec tv_[2])
+{
+  fuse_get_context()->private_data = fs_->user_data;
+
+  if(fs_->op.futimens)
+    return fs_->op.futimens(ffi_,tv_);
+
+  return -ENOSYS;
+}
+
 int fuse_fs_access(struct fuse_fs *fs, const char *path, int mask)
 {
 	fuse_get_context()->private_data = fs->user_data;
@@ -2256,73 +2302,11 @@ int fuse_fs_fallocate(struct fuse_fs *fs, const char *path, int mode,
 		return -ENOSYS;
 }
 
-static int is_open(struct fuse *f, fuse_ino_t dir, const char *name)
+static
+int
+is_node_open(const struct node *node_)
 {
-	struct node *node;
+  return (node_ && (node_->open_count > 0));
-	int isopen = 0;
-	pthread_mutex_lock(&f->lock);
-	node = lookup_node(f, dir, name);
-	if (node && node->open_count > 0)
-		isopen = 1;
-	pthread_mutex_unlock(&f->lock);
-	return isopen;
-}
-
-static char *hidden_name(struct fuse *f, fuse_ino_t dir, const char *oldname,
-			 char *newname, size_t bufsize)
-{
-	struct stat buf;
-	struct node *node;
-	struct node *newnode;
-	char *newpath;
-	int res;
-	int failctr = 10;
-
-	do {
-		pthread_mutex_lock(&f->lock);
-		node = lookup_node(f, dir, oldname);
-		if (node == NULL) {
-			pthread_mutex_unlock(&f->lock);
-			return NULL;
-		}
-		do {
-			f->hidectr ++;
-			snprintf(newname, bufsize, ".fuse_hidden%08x%08x",
-				 (unsigned int) node->nodeid, f->hidectr);
-			newnode = lookup_node(f, dir, newname);
-		} while(newnode);
-
-		res = try_get_path(f, dir, newname, &newpath, NULL, false);
-		pthread_mutex_unlock(&f->lock);
-		if (res)
-			break;
-
-		memset(&buf, 0, sizeof(buf));
-		res = fuse_fs_getattr(f->fs, newpath, &buf);
-		if (res == -ENOENT)
-			break;
-		free(newpath);
-		newpath = NULL;
-	} while(res == 0 && --failctr);
-
-	return newpath;
-}
-
-static int hide_node(struct fuse *f, const char *oldpath,
-		     fuse_ino_t dir, const char *oldname)
-{
-	char newname[64];
-	char *newpath;
-	int err = -EBUSY;
-
-	newpath = hidden_name(f, dir, oldname, newname, sizeof(newname));
-	if (newpath) {
-		err = fuse_fs_rename(f->fs, oldpath, newpath);
-		if (!err)
-			err = rename_node(f, dir, oldname, dir, newname, 1);
-		free(newpath);
-	}
-	return err;
 }
 
 static int mtime_eq(const struct stat *stbuf, const struct timespec *ts)
@@ -2619,30 +2603,42 @@ static void fuse_lib_getattr(fuse_req_t req, fuse_ino_t ino,
 	struct stat buf;
 	char *path;
 	int err;
+        struct node *node;
+        struct fuse_file_info ffi = {0};
+
+        if(fi == NULL)
+          {
+            pthread_mutex_lock(&f->lock);
+            node = get_node(f,ino);
+            if(node->is_hidden)
+              {
+                fi = &ffi;
+                fi->fh = node->hidden_fh;
+              }
+            pthread_mutex_unlock(&f->lock);
+          }
 
 	memset(&buf, 0, sizeof(buf));
 
-	if (fi != NULL && f->fs->op.fgetattr)
+        err = (((fi == NULL) || (f->fs->op.fgetattr == NULL)) ?
-		err = get_path_nullok(f, ino, &path);
+               get_path(f,ino,&path) :
-	else
+               get_path_nullok(f,ino,&path));
-		err = get_path(f, ino, &path);
+
 	if (!err) {
 		struct fuse_intr_data d;
 		fuse_prepare_interrupt(f, req, &d);
-		if (fi)
+
-			err = fuse_fs_fgetattr(f->fs, path, &buf, fi);
+                err = ((fi == NULL) ?
-		else
+                       fuse_fs_getattr(f->fs,path,&buf) :
-			err = fuse_fs_getattr(f->fs, path, &buf);
+                       fuse_fs_fgetattr(f->fs,path,&buf,fi));
+
 		fuse_finish_interrupt(f, req, &d);
 		free_path(f, ino, path);
 	}
-	if (!err) {
-		struct node *node;
 
+	if (!err) {
 		pthread_mutex_lock(&f->lock);
-		node = get_node(f, ino);
+                node = get_node(f, ino);
-		if (node->is_hidden && buf.st_nlink > 0)
-			buf.st_nlink--;
 		if (f->conf.auto_cache)
 			update_stat(node, &buf);
 		pthread_mutex_unlock(&f->lock);
@@ -2661,6 +2657,19 @@ int fuse_fs_chmod(struct fuse_fs *fs, const char *path, mode_t mode)
 		return -ENOSYS;
 }
 
+int
+fuse_fs_fchmod(struct fuse_fs              *fs_,
+               const struct fuse_file_info *ffi_,
+               const mode_t                 mode_)
+{
+  fuse_get_context()->private_data = fs_->user_data;
+
+  if(fs_->op.fchmod)
+    return fs_->op.fchmod(ffi_,mode_);
+
+  return -ENOSYS;
+}
+
 static void fuse_lib_setattr(fuse_req_t req, fuse_ino_t ino, struct stat *attr,
 			     int valid, struct fuse_file_info *fi)
 {
@@ -2668,34 +2677,56 @@ static void fuse_lib_setattr(fuse_req_t req, fuse_ino_t ino, struct stat *attr,
 	struct stat buf;
 	char *path;
 	int err;
+        struct node *node;
+        struct fuse_file_info ffi = {0};
+
+        if(fi == NULL)
+          {
+            pthread_mutex_lock(&f->lock);
+            node = get_node(f,ino);
+            if(node->is_hidden)
+              {
+                fi = &ffi;
+                fi->fh = node->hidden_fh;
+              }
+            pthread_mutex_unlock(&f->lock);
+          }
 
 	memset(&buf, 0, sizeof(buf));
-	if (valid == FUSE_SET_ATTR_SIZE && fi != NULL &&
+
-	    f->fs->op.ftruncate && f->fs->op.fgetattr)
+        err = ((fi == NULL) ?
-		err = get_path_nullok(f, ino, &path);
+               get_path(f,ino,&path) :
-	else
+               get_path_nullok(f, ino, &path));
-		err = get_path(f, ino, &path);
+
 	if (!err) {
 		struct fuse_intr_data d;
+
 		fuse_prepare_interrupt(f, req, &d);
-		err = 0;
+
+                err = 0;
 		if (!err && (valid & FUSE_SET_ATTR_MODE))
-			err = fuse_fs_chmod(f->fs, path, attr->st_mode);
+                  err = ((fi == NULL) ?
-		if (!err && (valid & (FUSE_SET_ATTR_UID | FUSE_SET_ATTR_GID))) {
+                         fuse_fs_chmod(f->fs, path, attr->st_mode) :
-			uid_t uid = (valid & FUSE_SET_ATTR_UID) ?
+                         fuse_fs_fchmod(f->fs, fi, attr->st_mode));
-				attr->st_uid : (uid_t) -1;
+
-			gid_t gid = (valid & FUSE_SET_ATTR_GID) ?
+		if (!err && (valid & (FUSE_SET_ATTR_UID | FUSE_SET_ATTR_GID)))
-				attr->st_gid : (gid_t) -1;
+                  {
-			err = fuse_fs_chown(f->fs, path, uid, gid);
+                    uid_t uid = (valid & FUSE_SET_ATTR_UID) ?
-		}
+                      attr->st_uid : (uid_t) -1;
-		if (!err && (valid & FUSE_SET_ATTR_SIZE)) {
+                    gid_t gid = (valid & FUSE_SET_ATTR_GID) ?
-			if (fi)
+                      attr->st_gid : (gid_t) -1;
-				err = fuse_fs_ftruncate(f->fs, path,
+
-							attr->st_size, fi);
+                    err = ((fi == NULL) ?
-			else
+                           fuse_fs_chown(f->fs, path, uid, gid) :
-				err = fuse_fs_truncate(f->fs, path,
+                           fuse_fs_fchown(f->fs, fi, uid, gid));
-						       attr->st_size);
+                  }
-		}
+
+		if (!err && (valid & FUSE_SET_ATTR_SIZE))
+                  {
+                    err = ((fi == NULL) ?
+                           fuse_fs_truncate(f->fs, path, attr->st_size) :
+                           fuse_fs_ftruncate(f->fs, path, attr->st_size, fi));
+                  }
 #ifdef HAVE_UTIMENSAT
 		if (!err && f->utime_omit_ok &&
 		    (valid & (FUSE_SET_ATTR_ATIME | FUSE_SET_ATTR_MTIME))) {
@@ -2716,7 +2747,9 @@ static void fuse_lib_setattr(fuse_req_t req, fuse_ino_t ino, struct stat *attr,
 			else if (valid & FUSE_SET_ATTR_MTIME)
 				tv[1] = attr->st_mtim;
 
-			err = fuse_fs_utimens(f->fs, path, tv);
+                        err = ((fi == NULL) ?
+                               fuse_fs_utimens(f->fs, path, tv) :
+                               fuse_fs_futimens(f->fs, fi, tv));
 		} else
 #endif
 		if (!err &&
@@ -2727,17 +2760,20 @@ static void fuse_lib_setattr(fuse_req_t req, fuse_ino_t ino, struct stat *attr,
 			tv[0].tv_nsec = ST_ATIM_NSEC(attr);
 			tv[1].tv_sec = attr->st_mtime;
 			tv[1].tv_nsec = ST_MTIM_NSEC(attr);
-			err = fuse_fs_utimens(f->fs, path, tv);
+			err = ((fi == NULL) ?
-		}
+                               fuse_fs_utimens(f->fs, path, tv) :
-		if (!err) {
+                               fuse_fs_futimens(f->fs, fi, tv));
-			if (fi)
-				err = fuse_fs_fgetattr(f->fs, path, &buf, fi);
-			else
-				err = fuse_fs_getattr(f->fs, path, &buf);
 		}
+
+                if (!err)
+                  err = ((fi == NULL) ?
+                         fuse_fs_getattr(f->fs, path, &buf) :
+                         fuse_fs_fgetattr(f->fs, path, &buf, fi));
+
 		fuse_finish_interrupt(f, req, &d);
 		free_path(f, ino, path);
 	}
+
 	if (!err) {
 		if (f->conf.auto_cache) {
 			pthread_mutex_lock(&f->lock);
@@ -2746,8 +2782,9 @@ static void fuse_lib_setattr(fuse_req_t req, fuse_ino_t ino, struct stat *attr,
 		}
 		set_stat(f, ino, &buf);
 		fuse_reply_attr(req, &buf, f->conf.attr_timeout);
-	} else
+	} else {
 		reply_err(req, err);
+        }
 }
 
 static void fuse_lib_access(fuse_req_t req, fuse_ino_t ino, int mask)
@@ -2859,17 +2896,23 @@ static void fuse_lib_unlink(fuse_req_t req, fuse_ino_t parent,
 	int err;
 
 	err = get_path_wrlock(f, parent, name, &path, &wnode);
+
 	if (!err) {
 		struct fuse_intr_data d;
 
 		fuse_prepare_interrupt(f, req, &d);
-		if (!f->conf.hard_remove && is_open(f, parent, name)) {
+
-			err = hide_node(f, path, parent, name);
+		if(is_node_open(wnode))
-		} else {
+                  {
-			err = fuse_fs_unlink(f->fs, path);
+                    err = fuse_fs_prepare_hide(f->fs, path, &wnode->hidden_fh, 0);
-			if (!err)
+                    if(!err)
-				remove_node(f, parent, name);
+                      wnode->is_hidden = 1;
-		}
+                  }
+
+                err = fuse_fs_unlink(f->fs, path);
+                if(!err && !wnode->is_hidden)
+                  remove_node(f, parent, name);
+
 		fuse_finish_interrupt(f, req, &d);
 		free_path_wrlock(f, parent, wnode, path);
 	}
@@ -2936,13 +2979,12 @@ static void fuse_lib_rename(fuse_req_t req, fuse_ino_t olddir,
 		struct fuse_intr_data d;
 		err = 0;
 		fuse_prepare_interrupt(f, req, &d);
-		if (!f->conf.hard_remove && is_open(f, newdir, newname))
+		if (is_node_open(wnode2))
-			err = hide_node(f, newpath, newdir, newname);
+                  err = fuse_fs_prepare_hide(f->fs, newpath, &wnode2->hidden_fh, 1);
 		if (!err) {
-			err = fuse_fs_rename(f->fs, oldpath, newpath);
+                  err = fuse_fs_rename(f->fs, oldpath, newpath);
-			if (!err)
+                  if (!err)
-				err = rename_node(f, olddir, oldname, newdir,
+                    err = rename_node(f, olddir, oldname, newdir, newname);
-						  newname, 0);
 		}
 		fuse_finish_interrupt(f, req, &d);
 		free_path2(f, olddir, newdir, wnode1, wnode2, oldpath, newpath);
@@ -2979,9 +3021,11 @@ static void fuse_do_release(struct fuse *f, fuse_ino_t ino, const char *path,
 			    struct fuse_file_info *fi)
 {
 	struct node *node;
-	int unlink_hidden = 0;
+        uint64_t fh;
+        int was_hidden;
 	const char *compatpath;
 
+        fh = 0;
 	if (path != NULL || f->nullpath_ok || f->conf.nopath)
 		compatpath = path;
 	else
@@ -2992,25 +3036,17 @@ static void fuse_do_release(struct fuse *f, fuse_ino_t ino, const char *path,
 	pthread_mutex_lock(&f->lock);
 	node = get_node(f, ino);
 	assert(node->open_count > 0);
-	--node->open_count;
+	node->open_count--;
-	if (node->is_hidden && !node->open_count) {
+        was_hidden = 0;
-		unlink_hidden = 1;
+	if (node->is_hidden && (node->open_count == 0)) {
-		node->is_hidden = 0;
+          was_hidden = 1;
+          node->is_hidden = 0;
+          fh = node->hidden_fh;
 	}
 	pthread_mutex_unlock(&f->lock);
 
-	if(unlink_hidden) {
+        if(was_hidden)
-		if (path) {
+          fuse_fs_free_hide(f->fs,fh);
-			fuse_fs_unlink(f->fs, path);
-		} else if (f->conf.nopath) {
-			char *unlinkpath;
-
-			if (get_path(f, ino, &unlinkpath) == 0)
-				fuse_fs_unlink(f->fs, unlinkpath);
-
-			free_path(f, ino, unlinkpath);
-		}
-	}
 }
 
 static void fuse_lib_create(fuse_req_t req, fuse_ino_t parent,
@@ -3045,8 +3081,10 @@ static void fuse_lib_create(fuse_req_t req, fuse_ino_t parent,
 	}
 	if (!err) {
 		pthread_mutex_lock(&f->lock);
-		get_node(f, e.ino)->open_count++;
+                struct node *n = get_node(f,e.ino);
+                n->open_count++;
 		pthread_mutex_unlock(&f->lock);
+
 		if (fuse_reply_create(req, &e, fi) == -ENOENT) {
 			/* The open syscall was interrupted, so it
 			   must be cancelled */
@@ -3121,7 +3159,8 @@ static void fuse_lib_open(fuse_req_t req, fuse_ino_t ino,
 	}
 	if (!err) {
 		pthread_mutex_lock(&f->lock);
-		get_node(f, ino)->open_count++;
+                struct node *n = get_node(f,ino);
+                n->open_count++;
 		pthread_mutex_unlock(&f->lock);
 		if (fuse_reply_open(req, fi) == -ENOENT) {
 			/* The open syscall was interrupted, so it
@@ -4306,7 +4345,6 @@ static const struct fuse_opt fuse_lib_opts[] = {
 static void fuse_lib_help(void)
 {
 	fprintf(stderr,
-"    -o hard_remove         immediate removal (don't hide files)\n"
 "    -o use_ino             let filesystem set inode numbers\n"
 "    -o readdir_ino         try to fill in d_ino in readdir\n"
 "    -o kernel_cache        cache files in kernel\n"
@@ -4608,16 +4646,11 @@ void fuse_destroy(struct fuse *f)
 		for (i = 0; i < f->id_table.size; i++) {
 			struct node *node;
 
-			for (node = f->id_table.array[i]; node != NULL;
+			for (node = f->id_table.array[i]; node != NULL; node = node->id_next)
-			     node = node->id_next) {
+                          {
-				if (node->is_hidden) {
+                            if (node->is_hidden)
-					char *path;
+                              fuse_fs_free_hide(f->fs,node->hidden_fh);
-					if (try_get_path(f, node->nodeid, NULL, &path, NULL, false) == 0) {
+                          }
-						fuse_fs_unlink(f->fs, path);
-						free(path);
-					}
-				}
-			}
 		}
 	}
 	for (i = 0; i < f->id_table.size; i++) {

man/mergerfs.1

@@ -1,7 +1,7 @@
 .\"t
 .\" Automatically generated by Pandoc 1.19.2.4
 .\"
-.TH "mergerfs" "1" "2019\-02\-22" "mergerfs user manual" ""
+.TH "mergerfs" "1" "2019\-03\-21" "mergerfs user manual" ""
 .hy
 .SH NAME
 .PP
@@ -110,12 +110,6 @@ than libfuse.
 While not a default it is recommended it be enabled so that linked files
 share the same inode value.
 .IP \[bu] 2
-\f[B]hard_remove\f[]: force libfuse to immedately remove files when
-unlinked.
-This will keep the \f[C]\&.fuse_hidden\f[] files from showing up but if
-software uses an opened but unlinked file in certain ways it could
-result in errors.
-.IP \[bu] 2
 \f[B]dropcacheonclose=true|false\f[]: when a file is requested to be
 closed call \f[C]posix_fadvise\f[] on it first to instruct the kernel
 that we no longer need the data and it can drop its cache.
@@ -395,6 +389,11 @@ acquired by \f[C]open\f[] or \f[C]create\f[].
 \f[C]readdir\f[] has no real need for a policy given the purpose is
 merely to return a list of entries in a directory.
 \f[C]statfs\f[]\[aq]s behavior can be modified via other options.
+That said many times the current FUSE kernel driver will not always
+provide the file handle when a client calls \f[C]fgetattr\f[],
+\f[C]fchown\f[], \f[C]fchmod\f[], \f[C]futimens\f[], \f[C]ftruncate\f[],
+etc.
+This means it will call the regular, path based, versions.
 .PP
 When using policies which are based on a branch\[aq]s available space
 the base path provided is used.
@@ -428,13 +427,14 @@ T}
 T{
 search
 T}@T{
-access, getattr, getxattr, ioctl, listxattr, open, readlink
+access, getattr, getxattr, ioctl (directories), listxattr, open,
+readlink
 T}
 T{
 N/A
 T}@T{
-fallocate, fgetattr, fsync, ftruncate, ioctl, read, readdir, release,
+fchmod, fchown, futimens, ftruncate, fallocate, fgetattr, fsync, ioctl
-statfs, write
+(files), read, readdir, release, statfs, write
 T}
 .TE
 .PP
@@ -607,6 +607,44 @@ T}@T{
 ff
 T}
 .TE
+.SS ioctl
+.PP
+When \f[C]ioctl\f[] is used with an open file then it will use the file
+handle which was created at the original \f[C]open\f[] call.
+However, when using \f[C]ioctl\f[] with a directory mergerfs will use
+the \f[C]open\f[] policy to find the directory to act on.
+.SS unlink
+.PP
+In FUSE there is an opaque "file handle" which is created by
+\f[C]open\f[], \f[C]create\f[], or \f[C]opendir\f[], passed to the
+kernel, and then is passed back to the FUSE userland application by the
+kernel.
+Unfortunately, the FUSE kernel driver does not always send the file
+handle when it theoretically could/should.
+This complicates certain behaviors / workflows particularly in the high
+level API.
+As a result mergerfs is currently doing a few hacky things.
+.PP
+libfuse2 and libfuse3, when using the high level API, will rename names
+to \f[C]\&.fuse_hiddenXXXXXX\f[] if the file is open when unlinked or
+renamed over.
+It does this so the file is still available when a request referencing
+the now missing file is made.
+This file however keeps a \f[C]rmdir\f[] from succeeding and can be
+picked up by software reading directories.
+.PP
+The change mergerfs has done is that if a file is open when an unlink or
+rename happens it will open the file and keep it open till closed by all
+those who opened it prior.
+When a request comes in referencing that file and it doesn\[aq]t include
+a file handle it will instead use the file handle created at
+unlink/rename time.
+.PP
+This won\[aq]t result in technically proper behavior but close enough
+for many usecases.
+.PP
+The plan is to rewrite mergerfs to use the low level API so these
+invasive libfuse changes are no longer necessary.
 .SS rename & link
 .PP
 \f[B]NOTE:\f[] If you\[aq]re receiving errors from software when files
@@ -1547,6 +1585,10 @@ Don\[aq]t add the \f[C]splice_*\f[] arguments or add
 This, however, is not guaranteed to work.
 .SS rm: fts_read failed: No such file or directory
 .PP
+NOTE: This is only relevant to mergerfs versions at or below v2.25.x and
+should not occur in more recent versions.
+See the notes on \f[C]unlink\f[].
+.PP
 Not \f[I]really\f[] a bug.
 The FUSE library will move files when asked to delete them as a way to
 deal with certain edge cases and then later delete that file when its
@@ -1564,9 +1606,7 @@ Using the \f[B]hard_remove\f[] option will make it so these temporary
 files are not used and files are deleted immedately.
 That has a side effect however.
 Files which are unlinked and then they are still used (in certain forms)
-will result in an error.
+will result in an error (ENOENT).
-.PP
-A fix is in the works for this.
 .SH FAQ
 .SS How well does mergerfs scale? Is it "production ready?"
 .PP
@@ -1655,13 +1695,32 @@ transfering your data.
 Setting \f[C]func.mkdir=epall\f[] can simplify managing path
 perservation for \f[C]create\f[].
 .SS Why was libfuse embedded into mergerfs?
-.PP
+.IP "1." 3
 A significant number of users use mergerfs on distros with old versions
 of libfuse which have serious bugs.
 Requiring updated versions of libfuse on those distros isn\[aq]t
 pratical (no package offered, user inexperience, etc.).
 The only practical way to provide a stable runtime on those systems was
-to "vendor" the library into the project.
+to "vendor" / embed the library into the project.
+.IP "2." 3
+mergerfs was written to use the high level API.
+There are a number of limitations in the HLAPI that make certain
+features difficult or impossible to implement.
+While some of these features could be patched into newer versions of
+libfuse without breaking the public API some of them would require hacky
+code to provide backwards compatibility.
+While it may still be worth working with upstream to address these
+issues in future versions, since the library needs to be vendored for
+stability and compatibility reasons it is preferable / easier to modify
+the API.
+Longer term the plan is to rewrite mergerfs to use the low level API.
+.SS Why did support for system libfuse get removed?
+.PP
+See above first.
+.PP
+If/when mergerfs is rewritten to use the low\-level API then it\[aq]ll
+be plausible to support system libfuse but till then its simply too much
+work to manage the differences across the versions.
 .SS Why use mergerfs over mhddfs?
 .PP
 mhddfs is no longer maintained and has some known stability and security
@@ -1819,7 +1878,10 @@ You could also set \f[C]xattr\f[] to \f[C]noattr\f[] or \f[C]nosys\f[]
 to short circuit or stop all xattr requests.
 .SS What are these .fuse_hidden files?
 .PP
-When not using \f[C]hard_remove\f[] libfuse will create
+NOTE: mergerfs >= 2.26.0 will not have these temporary files.
+See the notes on \f[C]unlink\f[].
+.PP
+When not using \f[B]hard_remove\f[] libfuse will create
 \&.fuse_hiddenXXXXXXXX files when an opened file is unlinked.
 This is to simplify "use after unlink" usecases.
 There is a possibility these files end up being picked up by software

src/fs_base_chmod.hpp

@@ -35,24 +35,6 @@ namespace fs
     return ::chmod(path_.c_str(),mode_);
   }
 
-  static
-  inline
-  int
-  fchmod(const int    fd_,
-         const mode_t mode_)
-  {
-    return ::fchmod(fd_,mode_);
-  }
-
-  static
-  inline
-  int
-  fchmod(const int          fd_,
-         const struct stat &st_)
-  {
-    return ::fchmod(fd_,st_.st_mode);
-  }
-
   static
   inline
   int
@@ -78,30 +60,4 @@ namespace fs
 
     return 0;
   }
-
-  static
-  inline
-  int
-  fchmod_check_on_error(const int          fd_,
-                        const struct stat &st_)
-  {
-    int rv;
-
-    rv = fs::fchmod(fd_,st_);
-    if(rv == -1)
-      {
-        int error;
-        struct stat tmpst;
-
-        error = errno;
-        rv = fs::fstat(fd_,&tmpst);
-        if(rv == -1)
-          return -1;
-
-        if((st_.st_mode & MODE_BITS) != (tmpst.st_mode & MODE_BITS))
-          return (errno=error,-1);
-      }
-
-    return 0;
-  }
 }

src/fs_base_chown.hpp

@@ -66,25 +66,6 @@ namespace fs
     return fs::lchown(path_,st_.st_uid,st_.st_gid);
   }
 
-  static
-  inline
-  int
-  fchown(const int   fd_,
-         const uid_t uid_,
-         const gid_t gid_)
-  {
-    return ::fchown(fd_,uid_,gid_);
-  }
-
-  static
-  inline
-  int
-  fchown(const int          fd_,
-         const struct stat &st_)
-  {
-    return fs::fchown(fd_,st_.st_uid,st_.st_gid);
-  }
-
   static
   inline
   int
@@ -111,31 +92,4 @@ namespace fs
 
     return 0;
   }
-
-  static
-  inline
-  int
-  fchown_check_on_error(const int          fd_,
-                        const struct stat &st_)
-  {
-    int rv;
-
-    rv = fs::fchown(fd_,st_);
-    if(rv == -1)
-      {
-        int error;
-        struct stat tmpst;
-
-        error = errno;
-        rv = fs::fstat(fd_,&tmpst);
-        if(rv == -1)
-          return -1;
-
-        if((st_.st_uid != tmpst.st_uid) ||
-           (st_.st_gid != tmpst.st_gid))
-          return (errno=error,-1);
-      }
-
-    return 0;
-  }
 }

src/fs_base_fchmod.hpp

@@ -0,0 +1,72 @@
+/*
+  ISC License
+
+  Copyright (c) 2019, Antonio SJ Musumeci <trapexit@spawn.link>
+
+  Permission to use, copy, modify, and/or distribute this software for any
+  purpose with or without fee is hereby granted, provided that the above
+  copyright notice and this permission notice appear in all copies.
+
+  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+  ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+  OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+*/
+
+#pragma once
+
+#include "fs_base_stat.hpp"
+
+#include <sys/stat.h>
+
+#define MODE_BITS (S_ISUID|S_ISGID|S_ISVTX|S_IRWXU|S_IRWXG|S_IRWXO)
+
+namespace fs
+{
+  static
+  inline
+  int
+  fchmod(const int    fd_,
+         const mode_t mode_)
+  {
+    return ::fchmod(fd_,mode_);
+  }
+
+  static
+  inline
+  int
+  fchmod(const int          fd_,
+         const struct stat &st_)
+  {
+    return ::fchmod(fd_,st_.st_mode);
+  }
+
+  static
+  inline
+  int
+  fchmod_check_on_error(const int          fd_,
+                        const struct stat &st_)
+  {
+    int rv;
+
+    rv = fs::fchmod(fd_,st_);
+    if(rv == -1)
+      {
+        int error;
+        struct stat tmpst;
+
+        error = errno;
+        rv = fs::fstat(fd_,&tmpst);
+        if(rv == -1)
+          return -1;
+
+        if((st_.st_mode & MODE_BITS) != (tmpst.st_mode & MODE_BITS))
+          return (errno=error,-1);
+      }
+
+    return 0;
+  }
+}

src/fs_base_fchown.hpp

@@ -0,0 +1,74 @@
+/*
+  ISC License
+
+  Copyright (c) 2019, Antonio SJ Musumeci <trapexit@spawn.link>
+
+  Permission to use, copy, modify, and/or distribute this software for any
+  purpose with or without fee is hereby granted, provided that the above
+  copyright notice and this permission notice appear in all copies.
+
+  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+  ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+  OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+*/
+
+#pragma once
+
+#include "fs_base_stat.hpp"
+
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+namespace fs
+{
+  static
+  inline
+  int
+  fchown(const int   fd_,
+         const uid_t uid_,
+         const gid_t gid_)
+  {
+    return ::fchown(fd_,uid_,gid_);
+  }
+
+  static
+  inline
+  int
+  fchown(const int          fd_,
+         const struct stat &st_)
+  {
+    return fs::fchown(fd_,st_.st_uid,st_.st_gid);
+  }
+
+  static
+  inline
+  int
+  fchown_check_on_error(const int          fd_,
+                        const struct stat &st_)
+  {
+    int rv;
+
+    rv = fs::fchown(fd_,st_);
+    if(rv == -1)
+      {
+        int error;
+        struct stat tmpst;
+
+        error = errno;
+        rv = fs::fstat(fd_,&tmpst);
+        if(rv == -1)
+          return -1;
+
+        if((st_.st_uid != tmpst.st_uid) ||
+           (st_.st_gid != tmpst.st_gid))
+          return (errno=error,-1);
+      }
+
+    return 0;
+  }
+}

src/fs_base_utime.hpp

@@ -45,15 +45,15 @@ namespace fs
   static
   inline
   int
-  utime(const int          fd_,
+  futime(const int          fd_,
-        const struct stat &st_)
+         const struct stat &st_)
   {
-    struct timespec times[2];
+    struct timespec ts[2];
 
-    times[0] = *fs::stat_atime(&st_);
+    ts[0] = *fs::stat_atime(&st_);
-    times[1] = *fs::stat_mtime(&st_);
+    ts[1] = *fs::stat_mtime(&st_);
 
-    return fs::utime(fd_,times);
+    return fs::futimens(fd_,ts);
   }
 
   static

src/fs_base_utime_generic.hpp

@@ -292,8 +292,8 @@ namespace fs
   static
   inline
   int
-  utime(const int             fd_,
+  futimens(const int             fd_,
-        const struct timespec ts_[2])
+           const struct timespec ts_[2])
   {
     int rv;
     struct timeval  tv[2];

src/fs_base_utime_utimensat.hpp

@@ -39,9 +39,9 @@ namespace fs
   static
   inline
   int
-  utime(const int             fd,
+  futimens(const int             fd_,
-        const struct timespec times[2])
+           const struct timespec ts_[2])
   {
-    return ::futimens(fd,times);
+    return ::futimens(fd_,ts_);
   }
 }

src/fs_clonefile.cpp

@@ -20,6 +20,8 @@
 #include "fs_base_chown.hpp"
 #include "fs_base_fadvise.hpp"
 #include "fs_base_fallocate.hpp"
+#include "fs_base_fchmod.hpp"
+#include "fs_base_fchown.hpp"
 #include "fs_base_ftruncate.hpp"
 #include "fs_base_stat.hpp"
 #include "fs_base_utime.hpp"
@@ -109,7 +111,7 @@ namespace fs
     if(rv == -1)
       return -1;
 
-    rv = fs::utime(dst_fd_,src_st);
+    rv = fs::futime(dst_fd_,src_st);
     if(rv == -1)
       return -1;
 

src/fuse_fchmod.cpp

@@ -0,0 +1,52 @@
+/*
+  Copyright (c) 2019, Antonio SJ Musumeci <trapexit@spawn.link>
+
+  Permission to use, copy, modify, and/or distribute this software for any
+  purpose with or without fee is hereby granted, provided that the above
+  copyright notice and this permission notice appear in all copies.
+
+  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+  ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+  OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+*/
+
+#include "errno.hpp"
+#include "fileinfo.hpp"
+#include "fs_base_fchmod.hpp"
+
+#include <fuse.h>
+
+#include <sys/stat.h>
+
+namespace l
+{
+  static
+  int
+  fchmod(const int    fd_,
+         const mode_t mode_)
+  {
+    int rv;
+
+    rv = fs::fchmod(fd_,mode_);
+    if(rv == -1)
+      return -errno;
+
+    return rv;
+  }
+}
+
+namespace FUSE
+{
+  int
+  fchmod(const struct fuse_file_info *ffi_,
+         const mode_t                 mode_)
+  {
+    FileInfo *fi = reinterpret_cast<FileInfo*>(ffi_->fh);
+
+    return l::fchmod(fi->fd,mode_);
+  }
+}

src/fuse_fchmod.hpp

@@ -0,0 +1,28 @@
+/*
+  Copyright (c) 2019, Antonio SJ Musumeci <trapexit@spawn.link>
+
+  Permission to use, copy, modify, and/or distribute this software for any
+  purpose with or without fee is hereby granted, provided that the above
+  copyright notice and this permission notice appear in all copies.
+
+  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+  ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+  OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+*/
+
+#pragma once
+
+#include <fuse.h>
+
+#include <sys/stat.h>
+
+namespace FUSE
+{
+  int
+  fchmod(const struct fuse_file_info *ffi_,
+         const mode_t                 mode_);
+}

src/fuse_fchown.cpp

@@ -0,0 +1,54 @@
+/*
+  Copyright (c) 2019, Antonio SJ Musumeci <trapexit@spawn.link>
+
+  Permission to use, copy, modify, and/or distribute this software for any
+  purpose with or without fee is hereby granted, provided that the above
+  copyright notice and this permission notice appear in all copies.
+
+  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+  ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+  OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+*/
+
+#include "errno.hpp"
+#include "fileinfo.hpp"
+#include "fs_base_fchown.hpp"
+
+#include <fuse.h>
+
+#include <unistd.h>
+
+namespace l
+{
+  static
+  int
+  fchown(const int   fd_,
+         const uid_t uid_,
+         const gid_t gid_)
+  {
+    int rv;
+
+    rv = fs::fchown(fd_,uid_,gid_);
+    if(rv == -1)
+      return -errno;
+
+    return rv;
+  }
+}
+
+namespace FUSE
+{
+  int
+  fchown(const struct fuse_file_info *ffi_,
+         const uid_t                  uid_,
+         const gid_t                  gid_)
+  {
+    FileInfo *fi = reinterpret_cast<FileInfo*>(ffi_->fh);
+
+    return l::fchown(fi->fd,uid_,gid_);
+  }
+}

src/fuse_fchown.hpp

@@ -0,0 +1,27 @@
+/*
+  Copyright (c) 2019, Antonio SJ Musumeci <trapexit@spawn.link>
+
+  Permission to use, copy, modify, and/or distribute this software for any
+  purpose with or without fee is hereby granted, provided that the above
+  copyright notice and this permission notice appear in all copies.
+
+  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+  ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+  OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+*/
+
+#pragma once
+
+#include <fuse.h>
+
+namespace FUSE
+{
+  int
+  fchown(const struct fuse_file_info *ffi_,
+         uid_t                        uid_,
+         gid_t                        gid_);
+}

src/fuse_free_hide.cpp

@@ -0,0 +1,37 @@
+/*
+  ISC License
+
+  Copyright (c) 2019, Antonio SJ Musumeci <trapexit@spawn.link>
+
+  Permission to use, copy, modify, and/or distribute this software for any
+  purpose with or without fee is hereby granted, provided that the above
+  copyright notice and this permission notice appear in all copies.
+
+  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+  ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+  OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+*/
+
+#include "fileinfo.hpp"
+#include "fs_base_close.hpp"
+
+#include <stdint.h>
+
+namespace FUSE
+{
+  int
+  free_hide(const uint64_t fh_)
+  {
+    FileInfo *fi = reinterpret_cast<FileInfo*>(fh_);
+
+    fs::close(fi->fd);
+
+    delete fi;
+
+    return 0;
+  }
+}

src/fuse_free_hide.hpp

@@ -0,0 +1,27 @@
+/*
+  ISC License
+
+  Copyright (c) 2019, Antonio SJ Musumeci <trapexit@spawn.link>
+
+  Permission to use, copy, modify, and/or distribute this software for any
+  purpose with or without fee is hereby granted, provided that the above
+  copyright notice and this permission notice appear in all copies.
+
+  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+  ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+  OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+*/
+
+#pragma once
+
+#include <stdint.h>
+
+namespace FUSE
+{
+  int
+  free_hide(const uint64_t fh_);
+}

src/fuse_futimens.cpp

@@ -0,0 +1,52 @@
+/*
+  Copyright (c) 2019, Antonio SJ Musumeci <trapexit@spawn.link>
+
+  Permission to use, copy, modify, and/or distribute this software for any
+  purpose with or without fee is hereby granted, provided that the above
+  copyright notice and this permission notice appear in all copies.
+
+  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+  ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+  OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+*/
+
+#include "errno.hpp"
+#include "fileinfo.hpp"
+#include "fs_base_utime.hpp"
+
+#include <fuse.h>
+
+#include <sys/stat.h>
+
+namespace l
+{
+  static
+  int
+  futimens(const int             fd_,
+           const struct timespec ts_[2])
+  {
+    int rv;
+
+    rv = fs::futimens(fd_,ts_);
+    if(rv == -1)
+      return -errno;
+
+    return rv;
+  }
+}
+
+namespace FUSE
+{
+  int
+  futimens(const struct fuse_file_info *ffi_,
+           const struct timespec        ts_[2])
+  {
+    FileInfo *fi = reinterpret_cast<FileInfo*>(ffi_->fh);
+
+    return l::futimens(fi->fd,ts_);
+  }
+}

src/fuse_futimens.hpp

@@ -0,0 +1,28 @@
+/*
+  Copyright (c) 2019, Antonio SJ Musumeci <trapexit@spawn.link>
+
+  Permission to use, copy, modify, and/or distribute this software for any
+  purpose with or without fee is hereby granted, provided that the above
+  copyright notice and this permission notice appear in all copies.
+
+  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+  ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+  OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+*/
+
+#pragma once
+
+#include <fuse.h>
+
+#include <sys/stat.h>
+
+namespace FUSE
+{
+  int
+  futimens(const struct fuse_file_info *ffi_,
+           const timespec               ts_[2]);
+}

src/fuse_prepare_hide.cpp

@@ -0,0 +1,44 @@
+/*
+  ISC License
+
+  Copyright (c) 2019, Antonio SJ Musumeci <trapexit@spawn.link>
+
+  Permission to use, copy, modify, and/or distribute this software for any
+  purpose with or without fee is hereby granted, provided that the above
+  copyright notice and this permission notice appear in all copies.
+
+  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+  ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+  OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+*/
+
+#include "fuse_open.hpp"
+
+#include <stdint.h>
+
+#include <stdio.h>
+
+namespace FUSE
+{
+  int
+  prepare_hide(const char *fusepath_,
+               uint64_t   *fh_,
+               int         type_)
+  {
+    int rv;
+    struct fuse_file_info ffi = {0};
+
+    ffi.flags = O_RDONLY|O_NOFOLLOW;
+    rv = FUSE::open(fusepath_,&ffi);
+    if(rv < 0)
+      return rv;
+
+    *fh_ = ffi.fh;
+
+    return 0;
+  }
+}

src/fuse_prepare_hide.hpp

@@ -0,0 +1,29 @@
+/*
+  ISC License
+
+  Copyright (c) 2019, Antonio SJ Musumeci <trapexit@spawn.link>
+
+  Permission to use, copy, modify, and/or distribute this software for any
+  purpose with or without fee is hereby granted, provided that the above
+  copyright notice and this permission notice appear in all copies.
+
+  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+  ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+  OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+*/
+
+#pragma once
+
+#include <stdint.h>
+
+namespace FUSE
+{
+  int
+  prepare_hide(const char *name_,
+               uint64_t   *fh_,
+               int         type_);
+}

src/mergerfs.cpp

@@ -25,12 +25,16 @@
 #include "fuse_create.hpp"
 #include "fuse_destroy.hpp"
 #include "fuse_fallocate.hpp"
+#include "fuse_fchmod.hpp"
+#include "fuse_fchown.hpp"
 #include "fuse_fgetattr.hpp"
 #include "fuse_flock.hpp"
 #include "fuse_flush.hpp"
+#include "fuse_free_hide.hpp"
 #include "fuse_fsync.hpp"
 #include "fuse_fsyncdir.hpp"
 #include "fuse_ftruncate.hpp"
+#include "fuse_futimens.hpp"
 #include "fuse_getattr.hpp"
 #include "fuse_getxattr.hpp"
 #include "fuse_init.hpp"
@@ -41,6 +45,7 @@
 #include "fuse_mknod.hpp"
 #include "fuse_open.hpp"
 #include "fuse_opendir.hpp"
+#include "fuse_prepare_hide.hpp"
 #include "fuse_read.hpp"
 #include "fuse_read_buf.hpp"
 #include "fuse_readdir.hpp"
@@ -77,6 +82,12 @@ namespace l
     ops.flag_nopath        = true;
     ops.flag_utime_omit_ok = true;
 
+    ops.prepare_hide = FUSE::prepare_hide;
+    ops.free_hide    = FUSE::free_hide;
+    ops.fchown       = FUSE::fchown;
+    ops.fchmod       = FUSE::fchmod;
+    ops.futimens     = FUSE::futimens;
+
     ops.access      = FUSE::access;
     ops.bmap        = NULL;
     ops.chmod       = FUSE::chmod;