github-mirror/mergerfs
2
0
Fork 0
mirror of https://github.com/trapexit/mergerfs.git synced 2024-11-29 20:13:57 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
1,312 Commits 30 Branches 95 Tags 26 MiB
fc4d14e487
Commit Graph

463 Commits

Author SHA1 Message Date
Antonio SJ Musumeci
76c8d48dbd Change threadpool to use concurrentqueue 2023-09-02 00:45:47 -05:00
Antonio SJ Musumeci
0c555e71a0 Initialize readdir threadpool after daemonizing 2023-08-28 22:47:04 -05:00
Antonio SJ Musumeci
8d7017273f Misc fixes for older platforms 2023-08-22 08:10:55 -05:00
Antonio SJ Musumeci
0ed03a1535 Use relaxed memory order for atomic counters 2023-08-21 15:43:03 -05:00
Antonio SJ Musumeci
fdecabfaab Update wyhash to final4 2023-08-21 00:18:41 -05:00
Antonio SJ Musumeci
e586d2fd1e Fix setting of ugids for concurrent readdir 2023-08-20 23:06:05 -05:00
Antonio SJ Musumeci
2377cb05b6 Add thread names for easier debugging 2023-08-14 17:51:41 -05:00
Antonio SJ Musumeci
0aafdefc18 Add ability to invalidate gid cache on demand 
SIGUSR2 or ioctl
 2023-08-13 23:30:52 -05:00
Antonio SJ Musumeci
6dcf6111af Add callbacks for newer functions 
* setupmapping
* removemapping
* syncfs
* tmpfile
 2023-08-13 17:38:13 -05:00
Antonio SJ Musumeci
c92a10021e Add readdir policies 2023-08-11 00:05:14 -05:00
Antonio SJ Musumeci
51d97bb444 Move fuse thread args out of fuse session object 2023-07-30 22:52:43 -05:00
Antonio SJ Musumeci
556c6522c0 Update fmt to v10.0.0 2023-07-15 17:47:36 -05:00
Antonio SJ Musumeci
543603bdfc Update ghc::filesystem to v1.5.14 2023-07-15 14:47:44 -05:00
Antonio SJ Musumeci
707d298d7c Create functions can set branches RO on EROFS 2023-07-13 21:19:13 -05:00
Antonio SJ Musumeci
fa537961c5 Simplify syslog wrapper a bit 2023-07-06 00:13:42 -05:00
Antonio SJ Musumeci
2722f4f64d Fix error handling for cached writes 2023-06-29 22:50:24 -05:00
Antonio SJ Musumeci
5ab0fbcaee Add manual GC triggering + configurable process queue depth 
Yes, these are unrelated changes but somehow ended up being
prototyped together and I'm too lazy to separate them.
 2023-06-26 22:44:23 -05:00
Antonio SJ Musumeci
6a14a10e6c Fix read/write behavior and return value depending on direct_io 
Also add parallel direct write option for 6.2+ kernels.
 2023-06-24 23:06:14 -05:00
Antonio SJ Musumeci
82430e1079 Fix moveonenospc when file opened with append mode 2023-06-12 23:40:08 -05:00
Antonio SJ Musumeci
c72997f298 Make srcmounts readonly 
All tools only read from srcmounts and the current setup had a bug
which caused branch mode and minfreespace to be stripped when using
config file.
 2023-04-05 22:24:24 -04:00
Antonio SJ Musumeci
cdcfef2ceb Remove duplicate entries in docs 2023-04-01 14:40:49 -04:00
Antonio SJ Musumeci
0a64d74b4b Change default read-thread-count from -1 to 0 as documented 2023-03-31 12:54:57 -04:00
Antonio SJ Musumeci
26fd11cb4d Workaround older gcc bug with namespacing std::hash 2023-03-28 21:53:38 -04:00
Antonio SJ Musumeci
eebd6e9eb4 Actively report to syslog when waiting for mounts 2023-03-28 19:01:21 -04:00
Antonio SJ Musumeci
1c5b0f2f5f Fix loading readonly / write once options from file 2023-03-19 23:40:32 -04:00
Ilya Taranov
d04e04e89a Fix config file parsing 2023-03-10 20:55:20 -08:00
Antonio SJ Musumeci
486c5d3734 Add ability to set readahead of mergerfs and branches 2023-03-09 00:12:57 -05:00
Antonio SJ Musumeci
373d331f39 Rework fuse read to use same buffers as messages 2023-02-26 23:31:44 -05:00
Antonio SJ Musumeci
84592a9f13 Remove splicing features 
After numerous tests it was found the splice features were at best
the same performance as standard IO and at worse actually slower.
To simplify the code all splice features are removed.
 2023-02-26 01:42:58 -05:00
Antonio SJ Musumeci
9effeb6869 Add "per-process" file caching feature 
Allows chosing page caching based on process name.
 2023-02-18 01:16:36 -05:00
Antonio SJ Musumeci
2d9c915641 Return NOTTY for all btrfs ioctl calls 
mergerfs tries to pass through ioctl calls but due to the complexity
of btrfs calls their is risk of segfaults. In the future it may be
required to whitelist all supported ioctl calls.
 2023-02-11 10:07:55 -05:00
Antonio SJ Musumeci
9adddfc66d Add option to lazy umount target mount point 2023-02-02 21:20:23 -05:00
Antonio SJ Musumeci
3e313cd605 Forbid adding mountpoint to branch list 2023-01-29 11:51:56 -05:00
Antonio SJ Musumeci
92ccfd3c89 Hard code setting of allow_other, ignore when set 2023-01-26 23:37:37 -05:00
Antonio SJ Musumeci
a7113bb163 Allow setting of scheduling priority 2023-01-25 23:38:31 -05:00
Antonio SJ Musumeci
939eb3996a Add option to wait for branches to become new mounts 
branches-mount-timeout=UINT64 in seconds (default: 0)
 2023-01-25 11:59:14 -05:00
Antonio SJ Musumeci
3ee93d4c56 Remove 'nonempty' argument 
This makes it like all other filesystems and brings it into alignment
with libfuse3's behavior.
 2023-01-16 23:33:36 -05:00
Antonio SJ Musumeci
d11807ef7c Remove use_ino option and make behavior same as if set 2023-01-16 16:21:44 -05:00
Antonio SJ Musumeci
24423b8d2a Add async message processing 2023-01-16 13:15:53 -05:00
Antonio SJ Musumeci
a6ca96fa08 Fix printf for 32bit systems 2022-12-08 22:04:15 -05:00
Antonio SJ Musumeci
7606430188 Remove libfuse abstraction in prep for adding request data 2022-11-26 23:30:27 -06:00
Antonio SJ Musumeci
32e2c9c48e Tweaks for 32bit systems 2022-11-05 23:28:10 -04:00
Antonio SJ Musumeci
3b0d7fb26e Fix setting of stat vars for symlinkify 2022-10-30 22:27:33 -04:00
Antonio SJ Musumeci
46ce28b149 Error when given invalid policy names 2022-08-06 23:01:23 -04:00
Andrea Gelmini
338e559e53
Fix typos 2022-06-22 11:51:23 +02:00
Antonio SJ Musumeci
c9a935835f Fix query of attr during symlink 
Using getattr when follow-symlink is enabled causes invalid type to the kernel
if symlink pointed to non-symlink.
 2022-03-19 16:18:46 -04:00
Antonio SJ Musumeci
c43b464d6c Call FUSE getattr rather than syscall for extra logic 
While more efficient calling regular getattr (stat) leads to incorrect
information being returned and would not work correctly when using
symlinkify.
 2021-12-18 17:07:23 -05:00
Antonio SJ Musumeci
e256c88e4c Get attrs for link, not target 2021-12-16 19:11:29 -05:00
Antonio SJ Musumeci
ee8532c70d Ensure server handling of locks is disabled 2021-11-27 00:01:12 -05:00
Antonio SJ Musumeci
18dead4d86 Add new debug printing routines 2021-10-27 22:12:10 -04:00
Antonio SJ Musumeci
5f737cb7bf Add option to log node memory usage metrics 2021-10-11 11:57:16 -04:00
Antonio SJ Musumeci
313aa2a1d3 Update ghc::filesystem and nonstd::optional 2021-09-30 19:43:05 -04:00
Antonio SJ Musumeci
8def16a811 Fix infinite loop in mfs action policy 2021-08-29 12:33:52 -04:00
Antonio SJ Musumeci
43a6d66e3c Major cleanup of libfuse to remove unneeded features 
* Remove request interrupt code. Required tracking of all requests unnecesssarily.
* Remove all debugging printing. Have plans to do full replacement.
* Remove deprecated functions.
* Remove unneeded error checking.
* Remove "userdata" which was unused.
* Remove allow_root feature.
 2021-08-29 00:05:30 -04:00
trapexit
3b8d495c43
Merge pull request #940 from trapexit/ioctl-runtime-removal 
Remove config ioctl calls
 2021-08-03 17:02:01 -04:00
Antonio SJ Musumeci
61c2187c14 Remove config ioctl calls 
While they were going to be the new way of interacting with the runtime config
I've decided to use toml based config for mergerfs v3. In that case all updates
will happen through the config file and ioctl will only be used to load/reload
the config rather than a key:value like API.
 2021-08-02 16:56:20 -04:00
trapexit
192e4a4af8
Merge pull request #939 from trapexit/update-nonstd-optional 
Update nonstd::optional to v3.4.0
 2021-08-02 16:29:40 -04:00
Antonio SJ Musumeci
7013ff9aeb Update nonstd::optional to v3.4.0 2021-08-02 16:10:56 -04:00
Antonio SJ Musumeci
3ce0dc56e4 Update ghc::filesystem to v1.5.8 2021-08-02 16:06:43 -04:00
Antonio SJ Musumeci
098f353bf1 pfrd: fix mod by zero error when all branches are filtered 2021-06-25 09:17:16 -04:00
Antonio SJ Musumeci
8adebc9489 new features: follow-symlinks, rename-exdev, link-exdev 
* follow-symlinks: allows mergerfs to transparently follow symlinks
* link-exdev: in the event a link returns EXDEV create a symlink instead
* rename-exdev: in the event a rename returns EXDEV move the oldpath and
  create a symlink for the newpath
 2021-02-11 23:24:08 -05:00
Antonio SJ Musumeci
d337574665 fix parsing of relative branch paths 2021-02-11 19:22:17 -05:00
trapexit
cbea2ba91b
Merge branch 'master' into readdir_rewind_fix 2021-02-09 18:13:08 -05:00
Antonio SJ Musumeci
f6e37672b8 reset dentry buffer when rewind'ed 2021-02-09 18:09:28 -05:00
Antonio SJ Musumeci
538467b86d config: rework global config, remove rwlock, make branches RCU like 
Also added unit tests. Should have done separately but found a number of bugs.
 2021-01-14 21:02:10 -05:00
Antonio SJ Musumeci
f50812a9ce dirname should not return an empty string 2020-12-18 16:04:17 -05:00
Antonio SJ Musumeci
44a98910ba msp policies stopped before root path 2020-12-17 19:55:49 -05:00
Antonio SJ Musumeci
7edd3c6cf3 config: fix invalid error check when parsing config file 2020-11-28 20:26:18 -05:00
Antonio SJ Musumeci
1b26f4908e general cleanup, slight memory reduction 2020-11-19 22:58:52 -05:00
Antonio SJ Musumeci
c3fffefbc1 msp policies: used wrong path to check existance 2020-11-19 18:34:28 -05:00
Antonio SJ Musumeci
04844420dc wyhash: use safety mode 1 
mode 0 is not intended for unaligned buffers
 2020-09-24 08:45:34 -04:00
Antonio SJ Musumeci
5a3184359d rework some function error handling 
chmod, chown, removexattr, setxattr, truncate, utimens:
* if no errors: return 0
* if no successes: return first error
* if file acted on was the same as related search function: return its value
* return 0

rmdir, unlink:
* if no errors: return 0
* return first error
 2020-09-23 00:01:25 -04:00
Antonio SJ Musumeci
610c75fa90 properly return const ref from tofrom string wrapper 2020-09-19 18:57:13 -04:00
Antonio SJ Musumeci
767039c492 option_parser: return 0 when requesting help or version 2020-09-10 21:58:23 -04:00
Antonio SJ Musumeci
fc3453932a branches: add per branch minfreespace w/ original value as default 
example: /mnt/disk0=RW,1G:/mnt/disk1=RW,2G
 2020-09-04 21:04:45 -04:00
Antonio SJ Musumeci
6311df774d optionally use lchmod depending on if on Linux or not (BSD) 2020-08-26 12:44:32 -04:00
Antonio SJ Musumeci
046844083f add {,ep,msp}pfrd policies 
Percentage Free Random Distribution

Chooses a random branch based on the available space percentage free.

IE: if branch A has 1G free and branch B has 2G then B should be chosen
twice as often.
 2020-08-25 22:19:01 -04:00
Antonio SJ Musumeci
7fd629ebf3 add #warning to make it more obvious what versions of functions are used 2020-08-18 19:28:36 -04:00
Antonio SJ Musumeci
2fe20b888e clean up and separate out fs_* files 2020-08-18 19:28:36 -04:00
Antonio SJ Musumeci
2696079601 break fs.hpp up into separate files 2020-08-18 19:28:36 -04:00
Antonio SJ Musumeci
ec15872a1f cleanup function signatures and definitions 2020-08-18 19:28:36 -04:00
Antonio SJ Musumeci
6cc6524997 change category to enum class 2020-08-18 19:28:36 -04:00
Antonio SJ Musumeci
0371b047f9 change from fasthash64 to wyhash 
wyhash has very good performance for small keys (our usecase) and
works on platforms with alignment concerns.

A user had an issue where fasthash64 lead to misaligned reads and
signal exceptions. wyhash does not have such issues and is faster.
 2020-08-06 19:00:05 -04:00
Antonio SJ Musumeci
30d13b7f77 inodecalc: add 32bit versions of hashs 2020-08-03 19:42:06 -04:00
Antonio SJ Musumeci
5989d41a86 readdir: use getdents64 for compatibility with ARM64 2020-08-02 18:43:09 -04:00
Antonio SJ Musumeci
0709b2e447 add missing options to usage 2020-07-30 16:55:18 -04:00
Antonio SJ Musumeci
15a0aedd25 cleanup: move some config data structures to separate files 2020-07-29 21:42:43 -04:00
Antonio SJ Musumeci
a93bd9f7fa freebsd: misc cleanups to get freebsd compiling 2020-07-29 13:08:38 -04:00
Antonio SJ Musumeci
139e61efaa nfsopenhack: remove empty file check 2020-07-26 17:11:06 -04:00
Antonio SJ Musumeci
93218a343a NFS open/creat hack 
A hack to work around non-POSIX NFS <-> FUSE behavior where it turns
an open(O_EXCL|O_CREAT,0444) into multiple calls that results in
EACCES.
 2020-07-23 19:36:10 -04:00
Antonio SJ Musumeci
3ec137c4ad policy: add "most shared path" policies 
Like path preserving but walks back the path till a match is found. Should
cover the usecase where someone wants a "less strict" form of path
preservation.
 2020-07-21 16:31:08 -04:00
Andrea Gelmini
3a6738475a Fix typos 2020-07-20 13:01:33 +02:00
Andrea Gelmini
468d42088a Removed duplicated include 2020-07-20 12:51:44 +02:00
Antonio SJ Musumeci
08d267fc61 moveonenospc: enhance the feature to allow using a policy 
Just like functions you can now set a policy for moveonfreenospc. This
allows for more flexibility.

For backwards compatibility moveonfreenospc=true is converted to
moveonfreenospc=mfs.

minfreespace does apply which is slightly different from original behavior.
 2020-07-15 19:33:05 -04:00
Antonio SJ Musumeci
c0990644e8 remove unnecessary libfuse flags nopath, nullpath_ok, and utime_omit_ok 2020-07-08 19:25:23 -04:00
Antonio SJ Musumeci
b4397f7f2e add 'inodecalc' option to allow selection of inode calculation algo 2020-06-28 20:02:07 -04:00
Antonio SJ Musumeci
dbdd3e22fc additional readdir refactor cleanup 
Differences between readdir and getdents is minimal at best. Leaving code
for now to allow for possible expansion later.
 2020-06-27 23:17:21 -04:00
Antonio SJ Musumeci
54c41c4994 rework config management 
* Add standard way to do str2val and val2str conversion
* Add support for a config file
* Add support for reading and writing settings via ioctl
 2020-06-18 18:54:25 -04:00
Antonio SJ Musumeci
698c4147fc fix getdent name length calculation 2020-03-12 23:45:20 -04:00
Antonio SJ Musumeci
3a46ec9fab fix filename hashing error 2020-03-07 12:09:40 -05:00
Antonio SJ Musumeci
9b2634a1e5 fix name length calculation for musc 2020-02-29 17:15:29 -05:00
Antonio SJ Musumeci
62873d2d3b use getdents64 on linux 2020-02-28 20:34:12 -05:00
Antonio SJ Musumeci
d119807adb restructure readdir, add readdir_plus 
Does not enable READDIR_AUTO. Might add in the future.
 2020-02-22 19:45:52 -05:00
Antonio SJ Musumeci
a646fe016d change inode conversion algo to reduce collision 2020-02-12 21:05:52 -05:00
Antonio SJ Musumeci
576ff3694e add cache.writeback to xattrs 2020-02-12 19:48:37 -05:00
Antonio SJ Musumeci
903d39f968 add writeback caching 2020-02-09 21:08:53 -05:00
Antonio SJ Musumeci
4d82ed9324 fix short writes on >2GB files when cloning file 2019-10-14 23:32:51 -04:00
Antonio SJ Musumeci
0fffabfbe3 only return 1 branch for rand/eprand policies 2019-09-07 17:25:06 -04:00
Antonio SJ Musumeci
50ad648ce4 initialize mutex to fix lockup 2019-09-07 14:10:36 -04:00
Antonio SJ Musumeci
4c4c27a93e set uid & gid when calling ioctl 
Turns out certain ioctl calls check the effective user id (FS_IOC_SETFLAGS).
 2019-06-17 13:37:15 -04:00
Antonio SJ Musumeci
5ca928e436 accept old arguments for backwards compatibility 2019-06-10 20:20:41 -04:00
Antonio SJ Musumeci
7cbd88ac81 allow setting of 'max_pages' (via 'fuse_msg_size') 
Linux 4.20 and above allow setting the number of pages per FUSE message
upto 256 (4K * 256 = 1MiB). This can greatly increase read and write
speeds depending on the workload.
 2019-06-03 21:01:19 -04:00
Antonio SJ Musumeci
8cb7195c3e add copy_file_range support 2019-06-03 07:12:05 -04:00
Antonio SJ Musumeci
529a953e30 add file caching across opens and runtime control 2019-05-30 00:10:08 -04:00
Antonio SJ Musumeci
ddf6a2f105 make async_read optional again 2019-05-28 15:57:50 -04:00
Antonio SJ Musumeci
2323c16316 add readdir caching 2019-05-23 22:11:33 -04:00
Antonio SJ Musumeci
1baa706d37 add symlink caching 2019-05-22 19:25:38 -04:00
Antonio SJ Musumeci
61cded5b34 fix for unlink race condition 2019-05-21 22:57:02 -04:00
Antonio SJ Musumeci
5f2221155c ensure parallel dirops is enabled if capable 2019-05-20 00:36:42 -04:00
Antonio SJ Musumeci
2b019b84b0 ensure async_aio is enabled if capable 2019-05-20 00:18:56 -04:00
trapexit
af1c0d78e8
Merge pull request #621 from trapexit/posix-acl 
add support for POSIX ACLs
 2019-05-19 18:26:49 -04:00
Antonio SJ Musumeci
80d56ac94e add support for POSIX ACLs 2019-05-19 18:17:55 -04:00
Antonio SJ Musumeci
1ca70521a4 remove 'remote' flock support 2019-05-19 17:34:56 -04:00
Antonio SJ Musumeci
e052446713 fix setting of fsname 2019-05-12 14:07:21 -04:00
Antonio SJ Musumeci
5883020784 tweak docs 2019-05-10 11:54:41 -04:00
Antonio SJ Musumeci
6ecc618d83 remove .fuse_hidden file creation 
addresses #521

mergerfs is written using libfuse's high level API. The HLAPI forces a certain
behavior with regard to unlinking or rename overwriting of open files. It
renames the file to .fuse_hiddenXXXXXX and when incoming requests refer
to the unlinked file it will use path based calls and use that path.

This behavior leads to rmdirs failing or some software noticing
the .fuse_hidden files.

This patch changes the HLAPI and removes entirely the .fuse_hidden behavior.

See the README for more.
 2019-04-13 21:41:51 -04:00
Antonio SJ Musumeci
2a075ead92 fall back to other file copy methods in clonefile 2019-04-13 15:08:13 -04:00
Antonio SJ Musumeci
b69819e4b2 ioctl: don't set outbufsz when not needed 2019-03-20 10:18:55 -04:00
Antonio SJ Musumeci
0918dfd117 make attr, entry, negative_entry cache timeouts runtime configurable 2019-02-23 01:03:20 -05:00
Antonio SJ Musumeci
95c0cc741d replace libfuse's autoconf with makefile 2019-02-18 21:36:32 -05:00
Antonio SJ Musumeci
7bf607bb6b fix FS_IOC_{GET,SET}{FLAGS,VERSION} ioctl calls 2019-02-17 11:48:19 -05:00
Antonio SJ Musumeci
940c323251 misc fixes for FreeBSD 2019-02-06 23:20:28 -05:00
Antonio SJ Musumeci
5ab7d2dd72 remove O_LARGEFILE 2019-02-06 21:06:11 -05:00
Antonio SJ Musumeci
9d9ee7b71d general code cleanup 2019-01-31 22:47:09 -05:00
Antonio SJ Musumeci
c5b2415daf remove defaults, hard code atomic_o_trunc, big_writes, and default_permissions 
`defaults` is a value used by all filesystems and isn't passed through to
mergerfs when mounting via the fstab or the mount command. This led
to inconsistent application of options. atomic_o_trunc, big_writes, and
default_permissions should be enabled all the time anyway and splice_*
can lead to issues so they are not always enabled.
 2019-01-30 22:46:21 -05:00
Antonio SJ Musumeci
5be7e007ce add statfs cache 2019-01-23 23:06:07 -05:00
Antonio SJ Musumeci
7a057daa0c add policy cache for 'open' 
A fusepath -> basepath cache for `open` to limit the overhead of FUSE in 'open, read/write, close' patterns (such as Transmission).
 2019-01-22 09:21:15 -05:00
Antonio SJ Musumeci
9fd3b968fc make ioctl on directories use open policy 2019-01-18 22:41:52 -05:00
Antonio SJ Musumeci
87b2795f2b set direct_io per open/create, now runtime configurable 2019-01-11 15:23:14 -05:00
Antonio SJ Musumeci
10f7f8bdea optimize link_cow eligibility check 2019-01-04 10:21:28 -05:00
Antonio SJ Musumeci
dfa1c1ad6f check minfreespace on newest policy create 2018-12-12 15:53:53 -05:00
Antonio SJ Musumeci
6ae68465cf fix building on alpine w/ musl 2018-11-28 17:08:40 -05:00
trapexit
3c1f19e85a
Merge pull request #534 from trapexit/xattr 
change xattr setting notsup to nosys
 2018-11-19 23:32:07 -05:00
Antonio SJ Musumeci
7d9458fdd1 change xattr setting notsup to nosys 2018-11-19 23:30:39 -05:00
Antonio SJ Musumeci
43b676a6e2 fix indexing of truncate targets 2018-11-19 23:25:16 -05:00
Antonio SJ Musumeci
c46134c03a fix building on platforms without O_PATH 2018-11-06 07:59:28 -05:00
Antonio SJ Musumeci
3631ab7df2 optimize readdir file dedup 
Use fasthash64 to hash filenames to uint64_t and store in khash
set. Significantly reduces malloc/free'ing and memory usage.
 2018-11-06 00:05:30 -05:00
Antonio SJ Musumeci
680f8194f9 add ability to change statfs behavior 2018-11-05 21:33:27 -05:00
Antonio SJ Musumeci
7524e57262 rename NW (no write) to NC (no create) 2018-11-01 23:37:45 -04:00
Antonio SJ Musumeci
b55ebba4ed add tagging branches RW/RO/NW 
This allows users to tag a branch as readonly or not for writing regardless
of how the filesystem is mounted. Should simplify deployments and offer
more flexibility.
 2018-11-01 23:13:49 -04:00