31 Commits

Author	SHA1	Message	Date
Bumsu Hyeon	4b99e84242	vfs/cache: fix handling of special characters in file names (#5875)	2022-01-13 13:23:25 +01:00
Nick Craig-Wood	d252816706	vfs: add vfs/stats remote control to show statistics - fixes #5816	2021-11-23 18:00:21 +00:00
Nick Craig-Wood	e43b5ce5e5	Remove github.com/pkg/errors and replace with std library version ... This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.	2021-11-07 11:53:30 +00:00
albertony	f3e71f129c	config: convert --cache-dir value to an absolute path	2021-10-11 15:08:35 +02:00
albertony	fbc7f2e61b	lib/file: improve error message when attempting to create dir on nonexistent drive on windows ... This replaces built-in os.MkdirAll with a patched version that stops the recursion when reaching the volume part of the path. The original version would continue recursion, and for extended length paths end up with \\? as the top-level directory, and the error message would then be something like: mkdir \\?: The filename, directory name, or volume label syntax is incorrect.	2021-10-01 23:18:39 +02:00
albertony	3a2f748aeb	vfs: ensure names used in cache path are legal on current os ... Fixes #5360	2021-08-19 20:14:50 +02:00
albertony	18be4ad10d	vfs: fix issue where empty dirs would build up in cache meta dir	2021-08-19 20:14:50 +02:00
Nick Craig-Wood	00a0ee1899	vfs: fix modtime changing when reading file into cache - fixes #5277 ... Before this change but after: aea8776a43 vfs: fix modtimes not updating when writing via cache #4763 When a file was opened read-only the modtime was read from the cached file. However this modtime wasn't correct leading to an incorrect result. This change fixes the definition of `item.IsDirty` to be true only when the data is dirty. This fixes the problem as a read only file isn't considered dirty.	2021-04-28 09:33:22 +01:00
Leo Luan	8f23cae1c0	vfs: Add cache reset for --vfs-cache-max-size handling at cache poll interval ... The vfs-cache-max-size parameter is probably confusing to many users. The cache cleaner checks cache size periodically at the --vfs-cache-poll-interval (default 60 seconds) interval and remove cache items in the following order. (1) cache items that are not in use and with age > vfs-cache-max-age (2) if the cache space used at this time still is larger than vfs-cache-max-size, the cleaner continues to remove cache items that are not in use. The cache cleaning process does not remove cache items that are currently in use. If the total space consumed by in-use cache items exceeds vfs-cache-max-size, the periodical cache cleaner thread does not do anything further and leaves the in-use cache items alone with a total space larger than vfs-cache-max-size. A cache reset feature was introduced in 1.53 which resets in-use (but not dirty, i.e., not being updated) cache items when additional cache data incurs an ENOSPC error. But this code was not activated in the periodical cache cleaning thread. This patch adds the cache reset step in the cache cleaner thread during cache poll to reset cache items until the total size of the remaining cache items is below vfs-cache-max-size.	2021-04-26 17:55:52 +01:00
Nick Craig-Wood	a4c4ddf052	vfs: rename files in cache and cancel uploads on directory rename ... Before this change rclone did not cancel an uploads or rename the cached files in the directory cache when a directory was renamed. This caused issues with uploads arriving in the wrong place on bucket based file systems. See: https://forum.rclone.org/t/after-a-directory-renmane-using-mv-files-are-not-visible-any-longer/22797	2021-03-22 09:07:01 +00:00
Nick Craig-Wood	aea8776a43	vfs: fix modtimes not updating when writing via cache - fixes #4763 ... This reads modtime from a dirty cache item if it exists. This mirrors the way reading the size works. This fixes the mod time not updating when the file is written, only when the writeback completes. See: https://forum.rclone.org/t/rclone-mount-and-changing-timestamps-after-writes/22629	2021-03-16 13:31:47 +00:00
albertony	459cc70a50	vfs: fix invalid cache path on windows when using :backend: as remote ... The initial ':' is included in the ad-hoc remote name, but is illegal character in Windows path. Replacing it with '^', which is legal in filesystems but illegal in regular remote names, so name conflict is avoided. Fixes #4544	2021-01-30 16:18:15 +00:00
Nick Craig-Wood	4f8ee736b1	vfs: make cache dir absolute before using it to fix path too long errors ... If --cache-dir is passed in as a relative path, then rclone will not be able to turn it into a UNC path under Windows, which means that file names longer than 260 chars will fail when stored in the cache. This patch makes the --cache-dir path absolute before using it. See: https://forum.rclone.org/t/handling-of-long-paths-on-windows-260-characters/20913	2020-12-11 10:00:51 +00:00
Nick Craig-Wood	2e21c58e6a	fs: deglobalise the config #4685 ... This is done by making fs.Config private and attaching it to the context instead. The Config should be obtained with fs.GetConfig and fs.AddConfig should be used to get a new mutable config that can be changed.	2020-11-26 16:40:12 +00:00
Nick Craig-Wood	d846210978	fs: Add context to NewFs #3257 #4685 ... This adds a context.Context parameter to NewFs and related calls. This is necessary as part of reading config from the context - backends need to be able to read the global config.	2020-11-09 18:05:54 +00:00
Josh Soref	d0888edc0a	Spelling fixes ... Fix spelling of: above, already, anonymous, associated, authentication, bandwidth, because, between, blocks, calculate, candidates, cautious, changelog, cleaner, clipboard, command, completely, concurrently, considered, constructs, corrupt, current, daemon, dependencies, deprecated, directory, dispatcher, download, eligible, ellipsis, encrypter, endpoint, entrieslist, essentially, existing writers, existing, expires, filesystem, flushing, frequently, hierarchy, however, implementation, implements, inaccurate, individually, insensitive, longer, maximum, metadata, modified, multipart, namedirfirst, nextcloud, obscured, opened, optional, owncloud, pacific, passphrase, password, permanently, persimmon, positive, potato, protocol, quota, receiving, recommends, referring, requires, revisited, satisfied, satisfies, satisfy, semver, serialized, session, storage, strategies, stringlist, successful, supported, surprise, temporarily, temporary, transactions, unneeded, update, uploads, wrapped Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-14 15:21:31 +01:00
Leo Luan	c5c56cda02	vfs: Add a missed update of used cache space ... The missed update can cause incorrect before-cleaning cache stats and a pre-mature condition broadcast in purgeOld before the cache space use is reduced below the quota.	2020-10-06 16:35:23 +01:00
Leo Luan	2295123cad	vfs: Add exponential backoff during ENOSPC retries ... Add an exponentially increasing delay during retries up ENOSPC error to avoid exhausting the 10 retries too soon when the cache space recovery from item resets is not available from the file system yet or consumed by other large cache writes.	2020-10-06 16:35:23 +01:00
Leo Luan	64d736a57b	vfs: Fix a race condition in retryFailedResets ... A failed item reset is saved in the errItems for retryFailedResets to process. If the item gets closed before the retry, the item may have been removed from the c.item array. Previous code did not account for this condition. This patch adds the check for the exitence of the retry items in retryFailedResets.	2020-10-06 16:35:23 +01:00
Hekmon	66def93373	mount cmd: update systemd status with cache stats	2020-10-06 16:21:30 +01:00
Sam Edwards	23b2c58018	vfs: Quiet removeNotInUse logging to debug when not removing	2020-09-02 11:55:20 +01:00
Leo Luan	c665201b85	vfs: support synchronous cache space recovery upon ENOSPC ... This patch provides the support of synchronous cache space recovery to allow read threads to recover from ENOSPC errors when cache space can be recovered from cache items that are not in use or safe to be reset/emptied . The patch complements the existing cache cleaning process in two ways. Firstly, the existing cache cleaning process is time-driven that runs periodically. The cache space can run out while the cache cleaner thread is still waiting for its next scheduled run. The io threads encountering ENOSPC return an internal error to the applications in this case even when cache space can be recovered to avoid this error. This patch addresses this problem by having the read threads kick the cache cleaner thread in this condition to recover cache space preventing unnecessary ENOSPC errors from being seen by the applications. Secondly, this patch enhances the cache cleaner to support cache item reset. Currently the cache purge process removes cache items that are not in use. This may not be sufficient when the total size of the working set exceeds the cache directory's capacity. Like in the current code, this patch starts the purge process by removing cache files that are not in use. Cache items whose access times are older than vfs-cache-max-age are removed first. After that, other not-in-use items are removed in LRU order until vfs-cache-max-size is reached. If the vfs-cache-max-size (the quota) is still not reached at this time, this patch adds a cache reset step to reset/empty cache files that are still in use but not dirtied. This enables application processes to continue without seeing an error even when the working set depletes the cache space as long as there is not a large write working set hoarding the entire cache space. By design this patch does not add ENOSPC error recovery for write IOs. Rclone does not empty a write cache item until the file data is written back to the backend upon close. Allowing more cache space to be consumed by dirty cache items when the cache space is already running low would increase the risk of exhausting the cache space in a way that the vfs mount becomes unreadable.	2020-08-25 21:12:06 +01:00
Nick Craig-Wood	b2f4f52b64	vfs cache: make logging consistent and remove some debug logging	2020-07-06 17:32:53 +01:00
Nick Craig-Wood	15402e46c9	vfs: Add recovered items on cache reload to directory listings ... Before this change, if we restarted an upload after a restart then the file would get uploaded but never added to the directory listings. This change makes sure we add virtual items to the directory cache when reloading the cache so that they show up properly.	2020-06-30 12:03:39 +01:00
Nick Craig-Wood	ee04732cbb	vfs: factor writeback and downloaders into their own packages	2020-06-30 12:03:39 +01:00
Nick Craig-Wood	fa472a340e	vfs: fix writeback deadlocks and other bugs ... - fix deadlock when cancelling upload - fix double upload and panic after cancelled upload - fix cancelation strategy of uploading files - don't cancel uploads if we don't modify the file - cancel uploads if we do modify the file - fix deadlock between Item and writeback - fix confusion about whether writeback item was being uploaded - fix cornercases in cancelling uploads and removing files	2020-06-30 12:01:36 +01:00
Nick Craig-Wood	58a7faa281	vfs: Make tests run reliably ... On file Remove - cancel any writebacks in progress - ignore error message deleting non existent file if file was in the process of being uploaded Writeback - Don't transfer the file if it has disappeared in the meantime - Take our own copy of the file name to avoid deadlocks - Fix delayed retry logic - Wait for upload to finish when cancelling upload Fix race condition in item saving Fix race condition in vfscache test Make sure we delete the file on the error path - this makes cascading failures much less likely	2020-06-30 12:01:36 +01:00
Nick Craig-Wood	496a87a665	vfs: restart pending uploads on restart of the cache	2020-06-30 12:01:36 +01:00
Nick Craig-Wood	e4e53a2e61	vfs: add --vfs-writeback option to delay writes back to cloud storage ... This is enabled by default and can be disabled with --vfs-writeback 0	2020-06-30 12:01:36 +01:00
Nick Craig-Wood	28255f1bac	vfs: fix errors when using > 260 char files in the cache in Windows ... This makes the cache use UNC paths on Windows. This stops the cache exploding when using > 260 character paths	2020-06-30 12:01:36 +01:00
Nick Craig-Wood	917cb4acb3	vfs: implement partial reads in --vfs-cache-mode full ... This allows reads to only read part of the file and it keeps on disk a cache of what parts of each file have been loaded. File data itself is kept in sparse files.	2020-06-30 12:01:36 +01:00