Commit Graph

381 Commits

Author SHA1 Message Date
Nick Craig-Wood
16d1da2c1e vfs: remove item.metaDirty as it was confusing and not used
See discussion in 
2021-04-28 09:33:22 +01:00
Nick Craig-Wood
00a0ee1899 vfs: fix modtime changing when reading file into cache - fixes
Before this change but after:

aea8776a43 vfs: fix modtimes not updating when writing via cache 

When a file was opened read-only the modtime was read from the cached
file. However this modtime wasn't correct leading to an incorrect
result.

This change fixes the definition of `item.IsDirty` to be true only
when the data is dirty. This fixes the problem as a read only file
isn't considered dirty.
2021-04-28 09:33:22 +01:00
albertony
2925e1384c Use binary prefixes for size and rate units
Includes adding support for additional size input suffix Mi and MiB, treated equivalent to M.
Extends binary suffix output with letter i, e.g. Ki and Mi.
Centralizes creation of bit/byte unit strings.
2021-04-27 02:25:52 +03:00
Leo Luan
8f23cae1c0 vfs: Add cache reset for --vfs-cache-max-size handling at cache poll interval
The vfs-cache-max-size parameter is probably confusing to many users.
The cache cleaner checks cache size periodically at the --vfs-cache-poll-interval
(default 60 seconds) interval and remove cache items in the following order.

(1) cache items that are not in use and with age > vfs-cache-max-age
(2) if the cache space used at this time still is larger than
vfs-cache-max-size, the cleaner continues to remove cache items that are
not in use.

The cache cleaning process does not remove cache items that are currently in use.
If the total space consumed by in-use cache items exceeds vfs-cache-max-size, the
periodical cache cleaner thread does not do anything further and leaves the in-use
cache items alone with a total space larger than vfs-cache-max-size.

A cache reset feature was introduced in 1.53 which resets in-use (but not dirty,
i.e., not being updated) cache items when additional cache data incurs an ENOSPC
error.  But this code was not activated in the periodical cache cleaning thread.

This patch adds the cache reset step in the cache cleaner thread during cache
poll to reset cache items until the total size of the remaining cache items is
below vfs-cache-max-size.
2021-04-26 17:55:52 +01:00
Nick Craig-Wood
2a40f00077 vfs: fix a code path which allows dirty data to be removed causing data loss
Before this change the VFS layer could remove a locally cached file
even if it had data which needed to be written back, thus causing data loss.

See: https://forum.rclone.org/t/rclone-1-55-doesnt-save-file-changes-if-the-file-has-been-reopened-during-upload-google-drive-mount/23646
2021-04-20 16:36:38 +01:00
Nick Craig-Wood
20e15e52a9 vfs: fix Create causing windows explorer to truncate files on CTRL-C CTRL-V
Before this fix, doing CTRL-C and CTRL-V on a file in Windows explorer
caused the **source** and the the destination to be truncated to 0.

This is because Windows opens the source file with Create with flags
`O_RDWR|O_CREATE|O_EXCL` but doesn't write to it - it only reads from
it. Rclone was taking the call to Create as a signal to always make a
new file, but this is incorrect.

This fix reads an existing file from the directory if it exists when
Create is called rather than always creating a new one. This fixes the
problem.

Fixes 
2021-03-31 14:48:02 +01:00
Nick Craig-Wood
b47d6001a9 vfs: fix directory renaming by renaming dirs cached in memory
Before this change, if a directory was renamed and it or any children
had virtual entries in it they weren't flushed.

The consequence of this was that the directory path got out sync with
the actual position of the directory in the tree, leading to listings
of the old directory rather than the new one.

The fix renames any directories remaining after the ForgetAll to have
the correct path which fixes the problem.

See: https://forum.rclone.org/t/after-a-directory-renmane-using-mv-files-are-not-visible-any-longer/22797
2021-03-22 09:07:01 +00:00
Nick Craig-Wood
a4c4ddf052 vfs: rename files in cache and cancel uploads on directory rename
Before this change rclone did not cancel an uploads or rename the
cached files in the directory cache when a directory was renamed.

This caused issues with uploads arriving in the wrong place on bucket
based file systems.

See: https://forum.rclone.org/t/after-a-directory-renmane-using-mv-files-are-not-visible-any-longer/22797
2021-03-22 09:07:01 +00:00
Nick Craig-Wood
c72d2c67ed vfs: add debug dump function to dump the state of the VFS cache 2021-03-22 09:06:44 +00:00
Nick Craig-Wood
5e95877840 vfs: fix modtime set if --vfs-cache-mode writes/full and no write
When using --vfs-cache-mode writes or full if a file was opened for
write intent, the modtime was set and the file was closed without
being modified the modtime would never be written back to storage.

The sequence of events

- app opens file with write intent
- app does set modtime
- rclone sets the modtime on the cache file, but not the remote file
  because it is open for write and can't be set yet
- app closes the file without changing it
- rclone doesn't upload the file because the file wasn't changed so
  the modtime doesn't get updated

This fixes the problem by making sure any unapplied modtime changes
are applied even if the file is not modified when being closed.

Fixes 
2021-03-16 13:36:48 +00:00
Nick Craig-Wood
8b491f7f3d vfs: fix modtimes changing by fractional seconds after upload
Before this change, rclone would return the modification times of the
cache file or the pending modtime which would be more accurate than
the modtime that the backend was capable of.

This meant that the modtime would be change slightly when the item was
actually uploaded.

For example modification times on Google Drive would be rounded to the
nearest millisecond.

This fixes the VFS layer to always return modtimes directly from an
object stored on the remote, or rounded to the precision that the
remote is capable of.
2021-03-16 13:31:47 +00:00
Nick Craig-Wood
aea8776a43 vfs: fix modtimes not updating when writing via cache - fixes
This reads modtime from a dirty cache item if it exists. This mirrors
the way reading the size works.

This fixes the mod time not updating when the file is written, only
when the writeback completes.

See: https://forum.rclone.org/t/rclone-mount-and-changing-timestamps-after-writes/22629
2021-03-16 13:31:47 +00:00
Nick Craig-Wood
c387eb8c09 vfs: don't set modification time if it was already correct
Before this change, rclone would set the modification time of an
object after it had been uploaded. However with --vfs-cache-mode
writes and above, the modification time of the object is already
correct as the cache backing file gets set with the correct
modification time before upload.

Setting the modification time causes another version to be created on
backends such as S3 so it should be avoided if possible.

This change checks to see if the modification time needs changing and
only sets it if necessary.

See: https://forum.rclone.org/t/produce-2-versions-when-overwrite-an-object-in-min-io/19634
2021-03-16 13:31:47 +00:00
Nick Craig-Wood
687a3b1832 vfs: fix data race discovered by the race detector
This fixes a place where we read from item.o without the item.mu held.
2021-03-15 19:22:07 +00:00
tYYGH
c0cf54067a
vfs: --vfs-used-is-size to report used space using recursive scan ()
Some backends, most notably S3, do not report the amount of bytes used.
This patch introduces a new flag that allows instead of relying on the
backend, use recursive scan similar to `rclone size` to compute the total
used space. However, this is ineffective and should be used as a last resort.

Co-authored-by: Yves G <theYinYeti@yalis.fr>
2021-02-17 23:36:13 +03:00
Nick Craig-Wood
9d37c208b7 vfs: document simultaneous usage with the same cache shouldn't be used
Fixes 
2021-02-16 17:15:05 +00:00
Nick Craig-Wood
ae3963e4b4 fs: Add string alternatives for setting options over the rc
Before this change options were read and set in native format. This
means for example nanoseconds for durations or an integer for
enumerated types, which isn't very convenient for humans.

This change enables these types to be set with a string with the
syntax as used in the command line instead, so `"10s"` rather than
`10000000000` or `"DEBUG"` rather than `8` for log level.
2021-02-07 14:56:41 +00:00
albertony
459cc70a50 vfs: fix invalid cache path on windows when using :backend: as remote
The initial ':' is included in the ad-hoc remote name, but is illegal character
in Windows path. Replacing it with '^', which is legal in filesystems but illegal
in regular remote names, so name conflict is avoided.

Fixes 
2021-01-30 16:18:15 +00:00
Martin Michlmayr
cd075f1703 docs: fix markup of arguments
Command line arguments have to be marked as code.
2021-01-25 22:40:46 +03:00
Nick Craig-Wood
b80d498304 vfs: fix file leaks with --vfs-cache-mode full and --buffer-size 0
Before this change using --vfs-cache-mode full and --buffer-size 0
together caused the vfs downloader to open more and more downloaders.

This is fixed by introducing a minimum size of 1M for the window to
look for an existing downloader.

Fixes 
2021-01-21 18:35:04 +00:00
albertony
9db51117dc vfs: docs: add note on os specific option 2020-12-28 13:59:34 +00:00
albertony
a9c9467210 vfs: docs: add note about use of --transfers in section about performance 2020-12-28 13:59:34 +00:00
albertony
e92cb9e8f8 mount: more user friendly mounting as network drive on windows
Add --network-mode option to activate mounting as network drive without having to set volume prefix.
Add support for automatic drive letter assignment (not specific to network drive mounting).
Allow full network share unc path in --volname, which will also implicitely activate network drive mounting.
Allow full network share unc path as mountpoint, which will also implicitely activate network drive mounting, and the specified path will be used as volume prefix and the remote will be mounted on an automatically assigned drive letter instead.
2020-12-28 13:59:34 +00:00
Nick Craig-Wood
4f8ee736b1 vfs: make cache dir absolute before using it to fix path too long errors
If --cache-dir is passed in as a relative path, then rclone will not
be able to turn it into a UNC path under Windows, which means that
file names longer than 260 chars will fail when stored in the cache.

This patch makes the --cache-dir path absolute before using it.

See: https://forum.rclone.org/t/handling-of-long-paths-on-windows-260-characters/20913
2020-12-11 10:00:51 +00:00
Nick Craig-Wood
2e21c58e6a fs: deglobalise the config
This is done by making fs.Config private and attaching it to the
context instead.

The Config should be obtained with fs.GetConfig and fs.AddConfig
should be used to get a new mutable config that can be changed.
2020-11-26 16:40:12 +00:00
Nick Craig-Wood
2347762b0d vfs: fix "file already exists" error for stale cache files
Before this change if a file was uploaded through a mount, then
deleted externally, trying to upload that file again could give EEXIST
"file already exists".

This was because the file already existing in the cache was confusing
rclone into thinking it already had the file.

The fix is to check that if rclone has a stale cache file then to
ignore it in this situation.

See: https://forum.rclone.org/t/rclone-cant-reuse-filenames/20400
2020-11-13 10:32:21 +00:00
Nick Craig-Wood
f980f230c5 vfs: fix virtual entries causing deleted files to still appear
Before this change, if a file was created on a remote but deleted
externally from that remote then there was potential for the delete to
never be noticed.

The sequence of events was:

- Create file on VFS - creates virtual directory entry
- File deleted externally to remote before the directory refreshed
- Now the file has a virtual add but is not in the listings so will never disappear

This patch fixes it by removing all virtual directory entries except
the following when the directory is re-read.

- On remotes which can't have empty directories: virtual directory
  adds are not flushed. These will remain virtual as long as the
  directory is empty.

- For virtual file add: files that are in the process of being
  uploaded are not flushed

This patch also adds the distinction between virtually added files and
directories.

It also refactors the virtual directory logic to make it easier to follow.

Fixes 
2020-11-10 16:47:25 +00:00
Nick Craig-Wood
1fb6ad700f accounting: add context.Context 2020-11-09 18:05:54 +00:00
Nick Craig-Wood
d69b96a94c test: Add context to mockfs.NewFs 2020-11-09 18:05:54 +00:00
Nick Craig-Wood
d846210978 fs: Add context to NewFs
This adds a context.Context parameter to NewFs and related calls.

This is necessary as part of reading config from the context -
backends need to be able to read the global config.
2020-11-09 18:05:54 +00:00
Nick Craig-Wood
43e0929339 vfs: fix vfs/refresh calls with fs= parameter
Before this change rclone gave an error when the fs parameter was
provided.

This change removes the fs parameter from the parameters once it has
been read which avoids the error.

See: https://forum.rclone.org/t/precaching-with-vfs-refresh-fails-with-an-error-when-having-multiple-cloud-drives/20267
2020-11-07 14:26:33 +00:00
Josh Soref
3e1cb8302a docs: spelling: etc.
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-28 18:16:23 +00:00
Josh Soref
e4a87f772f docs: spelling: e.g.
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-28 18:16:23 +00:00
Josh Soref
bbe7eb35f1 docs: spelling: server-side
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-28 18:16:23 +00:00
Josh Soref
d0888edc0a Spelling fixes
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-10-14 15:21:31 +01:00
Nick Craig-Wood
a86cedbc24 vfs: Fix --no-modtime to not attempt to set modtimes (as documented)
See: https://forum.rclone.org/t/rclone-mount-with-azure-blob-archive-tier/19414
2020-10-09 17:01:16 +01:00
Leo Luan
c5c56cda02 vfs: Add a missed update of used cache space
The missed update can cause incorrect before-cleaning cache stats
and a pre-mature condition broadcast in purgeOld before the cache
space use is reduced below the quota.
2020-10-06 16:35:23 +01:00
Leo Luan
2295123cad vfs: Add exponential backoff during ENOSPC retries
Add an exponentially increasing delay during retries up ENOSPC error
to avoid exhausting the 10 retries too soon when the cache space
recovery from item resets is not available from the file system yet
or consumed by other large cache writes.
2020-10-06 16:35:23 +01:00
Leo Luan
ff0280c0cb vfs: Fix missed concurrency control between some item operations and reset
Item reset is invoked by cache cleaner for synchronous recovery
from ENOSPC errors. The reset operation removes the cache file and
closes/reopens the downloaders.  Although most parts of reset and
other item operations are done with the item mutex held, the mutex
is released during fd.WriteAt and downloaders calls. We used preAccess
and postAccess calls to serialize Reset, ReadAt, and Open, but missed
some other item operations. The patch adds preAccess/postAccess
calls in Sync, Truncate, Close, WriteAt, and rename.
2020-10-06 16:35:23 +01:00
Leo Luan
64d736a57b vfs: Fix a race condition in retryFailedResets
A failed item reset is saved in the errItems for retryFailedResets
to process.  If the item gets closed before the retry, the item may
have been removed from the c.item array. Previous code did not
account for this condition. This patch adds the check for the
exitence of the retry items in retryFailedResets.
2020-10-06 16:35:23 +01:00
Leo Luan
5f1d5a1897 vfs: Fix a deadlock vulnerability in downloaders.Close
The downloaders.Close() call acquires the downloaders' mutex before
calling the wait group wait and the main downloaders thread has a
periodical (5 seconds interval) call to kick its waiters and the
waiter dispatch function tries to get the mutex. So a deadlock can
occur if the Close() call starts, gets the mutex, while the main
downloader thread already got the timer's tick and proceeded to
call kickWaiters. The deadlock happens when the Close call gets
the mutex between the timer's kick and the main downloader thread
gets the mutex first. So it's a pretty short period of time and
it probably explains why the problem has not surfaced, maybe
something like tens of nanoseconds out of 5 seconds (~10^^-8).
It took 5 days of continued stressing the Close calls for the
deadlock to appear.
2020-10-06 16:35:23 +01:00
Hekmon
66def93373 mount cmd: update systemd status with cache stats 2020-10-06 16:21:30 +01:00
Nick Craig-Wood
18ccf0f871 vfs: detect and recover from a file being removed externally from the cache
Before this change if a file was removed from the cache while rclone
is running then rclone would not notice and proceed to re-create it
full of zeros.

This change notices files that we expect to have data in going missing
and if they do logs an ERROR recovers.

It isn't recommended deleting files from the cache manually with
rclone running!

See: https://forum.rclone.org/t/corrupted-data-streaming-after-vfs-meta-files-removed/18997
Fixes 
2020-09-18 10:30:02 +01:00
Nick Craig-Wood
6a56ac1032 vfs,local: Log an ERROR if we fail to set the file to be sparse
See: https://forum.rclone.org/t/rclone-1-53-release/18880/73
2020-09-11 15:36:47 +01:00
Nick Craig-Wood
27b9ae4fc3 vfs: fix spurious error "vfs cache: failed to _ensure cache EOF"
Before this change the error message was produced for every file which
was confusing users.

After this change we check for EOF and return from ReadAt at that
point.

See: https://forum.rclone.org/t/rclone-1-53-release/18880/10
2020-09-03 10:25:00 +01:00
Nick Craig-Wood
3305079a03 vfs: fix typos in help 2020-09-02 14:24:44 +01:00
Sam Edwards
23b2c58018 vfs: Quiet removeNotInUse logging to debug when not removing 2020-09-02 11:55:20 +01:00
Nick Craig-Wood
70c8566cb8 fs: Pin created backends until parents are finalized
This attempts to solve the backend lifecycle problem by

- Pinning backends mentioned on the command line into the cache
  indefinitely

- Unpinning backends when the containing structure (VFS, wrapping
  backend) is destroyed

See: https://forum.rclone.org/t/rclone-rc-backend-command-not-working-as-expected/18834
2020-09-01 18:21:03 +01:00
Leo Luan
c665201b85 vfs: support synchronous cache space recovery upon ENOSPC
This patch provides the support of synchronous cache space recovery
to allow read threads to recover from ENOSPC errors when cache space
can be recovered from cache items that are not in use or safe to be
reset/emptied .

The patch complements the existing cache cleaning process in two ways.

Firstly, the existing cache cleaning process is time-driven that runs
periodically. The cache space can run out while the cache cleaner
thread is still waiting for its next scheduled run. The io threads
encountering ENOSPC return an internal error to the applications
in this case even when cache space can be recovered to avoid this
error. This patch addresses this problem by having the read threads
kick the cache cleaner thread in this condition to recover cache
space preventing unnecessary ENOSPC errors from being seen by the
applications.

Secondly, this patch enhances the cache cleaner to support cache
item reset. Currently the cache purge process removes cache
items that are not in use. This may not be sufficient when the
total size of the working set exceeds the cache directory's
capacity. Like in the current code, this patch starts the purge
process by removing cache files that are not in use. Cache items
whose access times are older than vfs-cache-max-age are removed first.
After that, other not-in-use items are removed in LRU order until
vfs-cache-max-size is reached. If the vfs-cache-max-size (the quota)
is still not reached at this time, this patch adds a cache reset
step to reset/empty cache files that are still in use but not
dirtied.  This enables application processes to continue without
seeing an error even when the working set depletes the cache space
as long as there is not a large write working set hoarding the
entire cache space.

By design this patch does not add ENOSPC error recovery for write
IOs. Rclone does not empty a write cache item until the file data
is written back to the backend upon close. Allowing more cache
space to be consumed by dirty cache items when the cache space is
already running low would increase the risk of exhausting the cache
space in a way that the vfs mount becomes unreadable.
2020-08-25 21:12:06 +01:00
Nick Craig-Wood
94a0991584 vfs: set the modtime of the cache file immediately
Before this change we set the modtime of the cache file when all
writers had finished.

This has the unfortunate effect that the file is uploaded with the
wrong modtime which means on backends which can't set modtimes except
when uploading files it is wrong.

This change sets the modtime of the cache file immediately in the
cache and in turn sets the modtime in the file info.
2020-08-20 16:24:04 +01:00
Nick Craig-Wood
d2ad293fae vfs: fix rename tests by waiting for writes to complete
Before this change the background writing of the file was racing with
the test of the object on the remote.

This meant that the tests passed locally but failed on a lot of the
remotes.
2020-08-19 17:04:17 +01:00
Nick Craig-Wood
6082096f7e vfs: check file exists in cache before renaming/setmodtime/deleting
Before this change we didn't check the file exists before renaming it,
setting its modification time or deleting it. If the file isn't in the
cache we don't need to do the action since it has been done on the
actual object, so these errors were producing unecessary log messages.

This change checks to see if the file exists first before doing those
actions.
2020-08-19 17:01:59 +01:00
Nick Craig-Wood
9a6fcd035b vfs: file: fix some locking issues reading f.d without the lock
Before this change we were reading Files.d without the lock. This
isn't allowed as d can change when the file is renamed.
2020-08-19 17:01:33 +01:00
Nick Craig-Wood
47d08ac1f1 vfs: recommend --vfs-cache-modes writes on backends which can't stream 2020-08-18 17:33:27 +01:00
Nick Craig-Wood
4d7f91309b vfs: fix download threads timing out
Before this fix, download threads would fill up the buffer and then
timeout even though data was still being read from them. If the client
was streaming slower than network speed this caused the downloader to
stop and be restarted continuously. This caused more potential for
skips in the download and unecessary network transactions.

This patch fixes that behaviour - as long as a downloader is being
read from more often than once every 5 seconds, it won't timeout.

This was done by:

- kicking the downloader whenever ensureDownloader is called
- making the downloader loop if it has already downloaded past the maxOffset
- making setRange() always kick the downloader
2020-08-06 17:26:18 +01:00
Nick Craig-Wood
109b695621 vfs: add --vfs-read-ahead parameter for use with --vfs-cache-mode full
This parameter causes extra read-ahead over --buffer-size which is not
buffered in memory but on disk.
2020-08-06 17:26:18 +01:00
Nick Craig-Wood
4441e012cf vfs: fix saving from chrome without --vfs-cache-mode writes
Due to Chrome's rather complicated use of file handles when saving
files from the download windows, rclone was attempting to truncate a
closed file.

The file appeared closed due to the handling of 0 length files.

This patch removes the check for the file being closed in the
WriteFileHandle.Truncate call. This is safe because the only action
this method takes is to emit an error message if the file is the wrong
size.

See: https://forum.rclone.org/t/google-drive-cannot-save-files-directly-from-browser-to-gdrive-mounted-path/17992/
2020-07-28 17:18:31 +01:00
Nick Craig-Wood
421585dd72 accounting: add context to Account and propagate changes
This is preparation for getting the Accounting to check the context,
buf first we need to get it in place. Since this is one of those
changes that makes lots of noise, this is in a seperate commit.
2020-07-28 16:41:17 +01:00
Nick Craig-Wood
0272a7f405 mount: change interface of mount commands to take mount options
This is in preparation of being able to pass mount options to the rc
command "mount/mount"
2020-07-24 10:48:51 +01:00
Nick Craig-Wood
2871268505 mount: change interface of mount commands to take VFS
This is in preparation of being able to pass options to the rc command
"mount/mount"
2020-07-23 12:30:41 +01:00
Nick Craig-Wood
ddfde68140 vfs: fix directory locking caused by slow directory listings
Before this fix we took the directory lock to read the ModTime of the
directory. This was causing locking on directories which were being
re-read from the backend.

This commit gives the modtime its own lock so it can be read even when
the directory is being updated.

See: https://forum.rclone.org/t/high-cpu-load-with-rclone-mount/17604
2020-07-18 09:08:18 +01:00
Nick Craig-Wood
126efaadcc vfs: fix renamed files not being uploaded with --vfs-cache-mode minimal
Before this change files that were in the cache and renamed with
--vfs-cache-mode minimal weren't renamed at all.

This fixes the problem and adds tests for all the different
combinations of cache modes and in and out of the cache.
2020-07-15 16:22:12 +01:00
Nick Craig-Wood
2adc057d95 vfs: fix very high load caused by slow directory listings
In this commit (released in v1.52.0)

6ca7198f mount: fix disappearing cwd problem

SetSys was introduced to cache node lookups.

Unfortunately taking the vfs.(*Dir) lock in SetSys causes any FUSE
operations on a directory to pile up behind slow directory listings.

In some situations this leads to very high load.

This commit fixes it by using atomic operations to read and write the
Sys value make it independent of the lock.

See: https://forum.rclone.org/t/high-cpu-load-with-rclone-mount/17604
See: 
2020-07-15 16:22:12 +01:00
Nick Craig-Wood
b2f4f52b64 vfs cache: make logging consistent and remove some debug logging 2020-07-06 17:32:53 +01:00
Nick Craig-Wood
c65ed26a7e vfs: vfscache: Fix renaming of items while they are being uploaded
Previous to the fix, if an item was being uploaded and it was renamed,
the upload would fail with missing checksum errors.

This change cancels any uploads in progress if the file is renamed.
2020-07-06 17:32:53 +01:00
Nick Craig-Wood
df5dbaf49b vfs: writeback: add Rename call for renaming items in the writeback queue 2020-07-06 17:32:53 +01:00
Nick Craig-Wood
63ebe4ca8d vfs: Reduce logging of metadata expiry to debug 2020-07-02 14:52:12 +01:00
Nick Craig-Wood
96c2fdb445 vfs: update VFS help 2020-06-30 12:03:39 +01:00
Nick Craig-Wood
8301a72453 vfs: Fix over downloading with --vfs-cache-mode full and --buffer-size 0
This was caused by the signal to stop buffering being ignored when
there was no buffer!

This is fixed by explicitly checking for no buffering and stopping.
2020-06-30 12:03:39 +01:00
Nick Craig-Wood
15402e46c9 vfs: Add recovered items on cache reload to directory listings
Before this change, if we restarted an upload after a restart then the
file would get uploaded but never added to the directory listings.

This change makes sure we add virtual items to the directory cache
when reloading the cache so that they show up properly.
2020-06-30 12:03:39 +01:00
Nick Craig-Wood
939860eb85 vfs: vfscache make TestCacheCleaner test more reliable 2020-06-30 12:03:39 +01:00
Nick Craig-Wood
530dc77cde vfs: Fix race condition in vfscache 2020-06-30 12:03:39 +01:00
Nick Craig-Wood
5db15cb157 vfs: make dir.ForgetAll and friends not forget virtual entries
Before this change dir.ForgetAll and vfs/forget would forget about
virtual directory entries.

This change preserves them.
2020-06-30 12:03:39 +01:00
Nick Craig-Wood
06a12f5e27 vfs: stop virtual directory entries dropping out of the directory cache
Rclone adds virtual directory entries to the directory cache when it
creates a file or directory.

Before this change these dropped out of the directory cache when the
directory cache was reloaded. This meant that when the directory cache
expired:

- On bucket based backends, empty directories would disappear
- When using VFS writeback, files in the process of uploading would disappear

This is fixed by keeping track of the virtual entries in each
directory. The virtual entries are removed when they become real - ie
the object is read back from the listing.

This also keeps tracks of deletes in the same way so if a file is
deleted, it will not re-appear when the directory cache is reloaded if
the deletion hasn't finished yet.
2020-06-30 12:03:39 +01:00
Nick Craig-Wood
143abe39f2 vfs: add tests for downloaders 2020-06-30 12:03:39 +01:00
Nick Craig-Wood
ee04732cbb vfs: factor writeback and downloaders into their own packages 2020-06-30 12:03:39 +01:00
Nick Craig-Wood
79455cc71e vfs: downloaders: remove unused osPath 2020-06-30 12:03:39 +01:00
Nick Craig-Wood
042e5fe097 vfs: downloader: limit the reader to 10 errors before giving up 2020-06-30 12:03:39 +01:00
Nick Craig-Wood
d273a9d82d vfs: remove items from writeback when dirty, don't just cancel the upload
This stops open items continually trying to be uploaded
2020-06-30 12:03:39 +01:00
Nick Craig-Wood
3eded3c4ac vfs: remove workaround Sleep() calls from tests 2020-06-30 12:03:39 +01:00
Nick Craig-Wood
ed32a759ed vfs: add test for writeBack.cancelUpload 2020-06-30 12:01:36 +01:00
Nick Craig-Wood
ef2d036884 vfs: make writeback heap sort in insertion order if expiry times equal
This makes the tests 100% consistent on platforms which have a lower
resolution timer like Windows.
2020-06-30 12:01:36 +01:00
Nick Craig-Wood
746c41f527 vfs: fix race in writeback tests 2020-06-30 12:01:36 +01:00
Nick Craig-Wood
b0fb457746 vfs: add tests for writeback 2020-06-30 12:01:36 +01:00
Nick Craig-Wood
b9ff495483 vfs: writeback - stop the timer explicitly on transfers exceeded 2020-06-30 12:01:36 +01:00
Nick Craig-Wood
8506066926 vfs: use call after functions in writeback to simplify code
This also fixes a bug in the uploader which didn't restart the timer
when the queue was empty.
2020-06-30 12:01:36 +01:00
Nick Craig-Wood
43018973ac vfs: decouple writeback from Item so it can be tested 2020-06-30 12:01:36 +01:00
Nick Craig-Wood
7e4ba54608 vfs: allow ReadAt and WriteAt to run concurrently with themselves
This should help with throughput on mounts and help when multiple
readers have the file open.

See: https://forum.rclone.org/t/concurrent-read-accesses-on-the-same-file-through-rclone-vfs-mount/17192
2020-06-30 12:01:36 +01:00
Nick Craig-Wood
2f66355f20 vfs: re-use existing VFS if possible 2020-06-30 12:01:36 +01:00
Nick Craig-Wood
7781ea8d59 vfs: add an optional fs parameter to vfs rc methods
Before this change we initialized the rc for a single VFS. However
rclone can have multiple VFSes in use now so this is no longer
adequate.

This change adds an optional fs parameter to all the VFS methods to
disambiguate VFSes when there is more than one in use.

It also adds a method vfs/list to show all the active VFSes.

This adds outline tests for the rc commands which didn't have tests
before.
2020-06-30 12:01:36 +01:00
Nick Craig-Wood
fa472a340e vfs: fix writeback deadlocks and other bugs
- fix deadlock when cancelling upload
- fix double upload and panic after cancelled upload
- fix cancelation strategy of uploading files
    - don't cancel uploads if we don't modify the file
    - cancel uploads if we do modify the file
- fix deadlock between Item and writeback
- fix confusion about whether writeback item was being uploaded
- fix cornercases in cancelling uploads and removing files
2020-06-30 12:01:36 +01:00
Nick Craig-Wood
279a516c53 vfs: add tests and fix bugs for vfscache.Item
Item
- Remove unused method getName
- Fix Truncate on unopened file
- Fix bug when downloading segments to fill out file on close
- Fix bug when WriteAt extends the file and we don't mark space as used

downloader
- Retry failed waiters every 5 seconds
2020-06-30 12:01:36 +01:00
Nick Craig-Wood
9ac5c6de14 vfs: cache: rework file downloader
- Download to multiple places at once in the stream
- Restart as necessary
- Timeout unused downloaders
- Close reader if too much data skipped
- Only use one file handle as use item for writing
- Implement --vfs-read-chunk-size and --vfs-read-chunk-limit
- fix deadlock between asyncbuffer and vfs cache downloader
- fix display of stream abandoned error which should be ignored
2020-06-30 12:01:36 +01:00
Nick Craig-Wood
58a7faa281 vfs: Make tests run reliably
On file Remove
- cancel any writebacks in progress
- ignore error message deleting non existent file if file was in the
  process of being uploaded

Writeback
- Don't transfer the file if it has disappeared in the meantime
- Take our own copy of the file name to avoid deadlocks
- Fix delayed retry logic
- Wait for upload to finish when cancelling upload

Fix race condition in item saving

Fix race condition in vfscache test

Make sure we delete the file on the error path - this makes cascading
failures much less likely
2020-06-30 12:01:36 +01:00
Nick Craig-Wood
496a87a665 vfs: restart pending uploads on restart of the cache 2020-06-30 12:01:36 +01:00
Nick Craig-Wood
e4e53a2e61 vfs: add --vfs-writeback option to delay writes back to cloud storage
This is enabled by default and can be disabled with --vfs-writeback 0
2020-06-30 12:01:36 +01:00
Nick Craig-Wood
28255f1bac vfs: fix errors when using > 260 char files in the cache in Windows
This makes the cache use UNC paths on Windows. This stops the cache
exploding when using > 260 character paths
2020-06-30 12:01:36 +01:00
Nick Craig-Wood
917cb4acb3 vfs: implement partial reads in --vfs-cache-mode full
This allows reads to only read part of the file and it keeps on disk a
cache of what parts of each file have been loaded.

File data itself is kept in sparse files.
2020-06-30 12:01:36 +01:00
Rob Calistri
b2ded6212b vfs: Change modtime of file before upload to current
Previously files before they get uploaded will inherit the directory modtime.  This changes that to use the current time instead.
2020-06-10 12:10:50 +01:00
Nick Craig-Wood
88df5927f9 vfs: funnel all read/write calls through ReadAt/WriteAt
This is in preparation for partial reads for read/write files
2020-06-09 18:07:41 +01:00
Nick Craig-Wood
8c37262e05 vfs: don't use embedded methods for read/write handles for clarity 2020-06-09 18:07:23 +01:00
Nick Craig-Wood
05bc19c331 vfs: Remove uneeded locking from read write handle String() 2020-06-09 18:04:50 +01:00
Nick Craig-Wood
50e31c6636 vfs: fix OS vs Unix path confusion - fixes ChangeNotify on Windows
See: https://forum.rclone.org/t/windows-mount-polling-not-recognising-all-changes-made-by-another-box/16708
2020-06-03 17:05:58 +01:00
Nick Craig-Wood
1bd3365868 vfs: fix TestVFSStatfs with known total, used and unknown free 2020-05-25 16:46:56 +01:00
Martin Michlmayr
041b201abd doc: fix typos throughout docs and code 2020-05-25 11:23:58 +01:00
Martin Michlmayr
a36ef8582f doc: use consistent capitalization 2020-05-20 15:54:51 +01:00
Martin Michlmayr
4aee962233 doc: fix typos throughout docs and code 2020-05-20 15:54:51 +01:00
Nick Craig-Wood
951099dbed vfs: change default --vfs-read-wait to 20ms
In my testing with local and remote storage this is a good compromise
between delaying the seeks and failing to wait for in sequence reads.

See: https://forum.rclone.org/t/constantly-high-iowait-add-log/14156/40
2020-05-18 18:09:23 +01:00
Nick Craig-Wood
0f9267d5fc vfs: factor waiting code from read and writes into common function 2020-05-18 18:09:23 +01:00
Nick Craig-Wood
3de9bd9d04 vfs: fix hang in read wait code - Fixes
Before this fix, rclone would sometimes hang in vfs.readAt().

This was due to a race condition causing rclone to miss the timeout
signal.

This was fixed by a small amount of extra locking.

This very likely also fixes a number of "failed to wait for
in-sequence read" errors.
2020-05-18 18:09:23 +01:00
Nick Craig-Wood
1a8c5708c5 vfs: ignore file not found errors from Hash in Read.Release
There is nothing we can do about this at this point and this error can
happen when moving files so we ignore it to clean the logs up.
2020-05-04 12:18:28 +01:00
Nick Craig-Wood
cfcdc85b26 vfs: Add SetSys() methods to Node to allow caching stuff on a node 2020-05-04 11:38:07 +01:00
Nick Craig-Wood
70db13e6e8 vfs: pin the Fs in use in the Fs cache
This means we can reliably look up the Fs from the cache when using
`backend/command`.
2020-05-01 17:11:45 +01:00
Nick Craig-Wood
77fb3c2511 vfs: bring DO NOT EDIT comments in line with "go help generate" 2020-04-30 12:24:44 +01:00
Nick Craig-Wood
86eaf43b00 vfs: fix tests for Statfs when running on backends with unknowns
This was broken in da41db4712
2020-04-22 18:14:03 +01:00
Nick Craig-Wood
eed9c5738d vfs: factor the vfs cache into its own package 2020-04-20 10:42:33 +01:00
Nick Craig-Wood
fd39cbc193 vfstests: move functional tests from mountlib and make them work with VFS
The tests are now run for the mount commands and for the plain VFS.

This makes the tests much easier to debug when running with a VFS than
through a mount.
2020-04-20 10:42:33 +01:00
Nick Craig-Wood
0961763082 vfs: add utility methods to match os package 2020-04-19 15:40:55 +01:00
Nick Craig-Wood
07908f3f54 vfs: bring open_tests.go generator back into line with the generated tests
In

54deb01f00 vfs: Make OpenFile and friends return EINVAL if O_RDONLY and O_TRUNC

The generated file open_test.go was edited directly without editing
the generator.

This commit brings the generator make_open_tests.go back into line
with that edit. It also makes it so `go generate` can be used to
regenerate the tests.
2020-04-19 15:18:49 +01:00
Nick Craig-Wood
1f50b70919 vfs: consistently use f.Path() or f._path() in File logs to avoid deadlocks
Previously we were using f which calls f.String() which calls f.Path()
which can cause a deadlock if uses carelessly.

This patch explicitly calls f.Path() or f._path() to bring attention
to the fact that there is a call to a method.
2020-04-19 15:16:43 +01:00
Nick Craig-Wood
19db0df639 vfs: stop reading Dir members from outside dir.go 2020-04-19 15:16:43 +01:00
Nick Craig-Wood
238f26cc90 vfs: stop reading File members from outside file.go
This also fixes locking for ReadFileHandle and WriteFileHandle
accessing File members
2020-04-19 15:16:43 +01:00
Nick Craig-Wood
268fcbb973 vfs: implement lock ordering between File and Dir to eliminate deadlocks
As part of this we take a copy of the directory path as calling
d.Path() violates the total locking order.

See the comment at the top of file.go for details
2020-04-19 15:16:43 +01:00
Nick Craig-Wood
da41db4712 vfs,mount,cmount: report 1PB free for unknown disk sizes
Factor the logic into the VFS layer so we don't have to duplicate it
into mount and cmount.

See: https://forum.rclone.org/t/rclone-mount-question/15454/
2020-04-11 13:31:10 +01:00
Nick Craig-Wood
3911a49256 vfs: make File lock and Dir lock not overlap to avoid deadlock
This was caused by this commit which wasn't part of 1.51.0

3c91abce74 vfs: fix race condition caused by unlocked reading of Dir.path
2020-04-02 21:14:45 +01:00
Yves G
b1b5e09081 vfs: make df output more consistent on a rclone mount.
When 2 values are known among vfs:{free,used,total}, compute the 3rd
2020-03-01 08:54:07 +00:00
Nick Craig-Wood
393b94bb70 vfs: add --vfs-read-wait and --vfs-write-wait flags
--vfs-read-wait duration    Time to wait for in-sequence read before seeking. (default 5ms)
    --vfs-write-wait duration   Time to wait for in-sequence write before giving error. (default 1s)

See: https://forum.rclone.org/t/constantly-high-iowait-add-log/14156
2020-02-27 16:12:33 +00:00
Nick Craig-Wood
3c91abce74 vfs: fix race condition caused by unlocked reading of Dir.path 2020-02-27 15:50:41 +00:00
Lars Lehtonen
4e1a511f88
vfs: explicitly ignore unused variables 2020-02-12 11:20:54 +00:00
Nick Craig-Wood
102a38bb95 docs: restore lost VFS poll interval docs
These came from 3d475dc0ee which unfortunately got added the
docs to the auto generated files.
2020-02-10 15:29:39 +00:00
Nick Craig-Wood
7a54e13110 docs: restore lost VFS case insensitive docs
These came from 1c4e33d4ad which unfortunately
added the docs to the auto generated files.
2020-02-10 15:29:39 +00:00
Nick Craig-Wood
84191ac6dc vfs: fix incorrect modtime for mv into mount with --vfs-cache-modes write
When a file has its modtime set while it is open we delay setting the
modtime until the file is closed.

The file is then uploaded in Flush. In Release we check the cached
file has been uploaded by comparing modtimes and or hashes and upload
it again if it has changed.

Before this change we forgot to change the time on the cached file
when we updated the time file on the object, so this mean that Release
reset the time to the wrong time and uploaded the file again on
remotes which don't support hashes (eg crypt).

The fix was to set the modtime of the cached file at the same time we
set the modtime of the remote object. This means that the files check
as identical in Release so it doesn't try to upload the file.

This means that we avoid a double upload and the modtime is correct.

See: https://forum.rclone.org/t/modification-time-with-vfs-cache/13906/8
2020-01-19 12:52:48 +00:00
Nick Craig-Wood
95c83b37fb vfs: only run TestRWCacheRename on remotes which can rename
This fixes the 1fichier integration tests.
2020-01-15 16:25:04 +00:00
Nick Craig-Wood
63128834da vfs: fix open file renaming on drive when using --vfs-cache-mode writes
Before this change, when uploading files from the VFS cache which were
pending a rename, rclone would use the new path of the object when
specifiying the destination remote.  This didn't cause a problem with
most backends as the subsequent rename did nothing, however with the
drive backend, since it updates objects, the incorrect Remote was
embedded in the object.  This caused the rename to apparently succeed
but the object be at the wrong location.

The fix for this was to make sure we upload to the path stored in the
object if available.

This problem was spotted by the new rename tests for the VFS layer.
2020-01-13 17:37:54 +00:00
Nick Craig-Wood
1b89b38a4c vfs: skip rename tests on remotes which can't rename 2020-01-05 12:34:47 +00:00
Nick Craig-Wood
1ab4985046 vfs: when renaming files in the cache, rename the cache item in memory too 2019-12-12 13:31:10 +00:00
Nick Craig-Wood
6e683b4359 vfs: fix rename of open files when using the VFS cache
Before this change, renaming an open file when using the VFS cache was
delayed until the file was closed.  This meant that the file was not
readable after a rename even though it is was in the cache.

After this change we rename the local cache file and the in memory
cache, delaying only the rename of the file in object storage.

See: https://forum.rclone.org/t/xen-orchestra-ebadf-bad-file-descriptor-write/13104
2019-12-12 13:31:10 +00:00
Nick Craig-Wood
241921c786 vfs: don't cache the path in RW file objects to fix renaming 2019-12-12 13:31:10 +00:00
Nick Craig-Wood
420ae905b5 vfs: make sure existing files opened for write show correct size
Before this change if an existing file was opened for write without
truncate its size would show as 0 rather than the full size of the
file.
2019-11-25 11:31:44 +00:00
Nick Craig-Wood
b5bb4c2a21 vfs: fix tests not to upload a 0 length file
Some remotes can't upload 0 length files, so this fixes the
TestCacheRename test so that it writes something to the file.
2019-11-15 09:26:40 +00:00
Nick Craig-Wood
358f5a8084 vfs: fix edge cases when reading ModTime from file
This fixes the unreliable test TestMount/CacheMode=full/TestFileModTime
2019-11-11 16:20:28 +00:00
Nick Craig-Wood
51efb349ac vfs: revise locking in file and dir to fix race conditions 2019-11-11 16:20:27 +00:00
Nick Craig-Wood
5beeac7959 vfs: make ReadAt for non cached files work better with non-sequential reads
This makes ReadAt for non cached files wait a short time (up to 5mS)
if it gets an out of order read (which would normally cause a seek and
which take a long time) to see if the gap will be filled with an in
order read.

This makes mount2 based on go-fuse work more efficiently and enables
async reading in normal mount.

A similar change was done for WriteAt in af030f74f5
2019-11-11 16:20:27 +00:00
Nick Craig-Wood
be5392f448 vfs: only calculate one hash for reads
This speeds up mounting on the local backend enormously.
2019-11-11 16:20:27 +00:00
Nick Craig-Wood
6150ae89d6 vfs: add a newly created file straight into the directory 2019-11-11 15:20:09 +00:00
Brett Dutro
53d55ae760 Add test for cache renaming functionality 2019-11-10 11:58:46 +00:00
Anagh Kumar Baranwal
5928704e1b On rename, rename in cache too if the file exists
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2019-11-10 11:58:46 +00:00
Brett Dutro
378a3f4133 mount: replace use of WriteAt with Write for cache mode >= writes and O_APPEND
os.File.WriteAt returns an error if a file was opened with O_APPEND.
This replaces it with os.File.Write if the file was opened with
O_APPEND.
2019-10-26 17:27:52 +01:00
Nick Craig-Wood
76f5e273d2 vfs: stop change notify polling clearing so much of the directory cache
Before this change, change notify polls would clear the directory
cache recursively. So uploading a file to the root would clear the
entire directory cache.

After this change we just invalidate the directory cache of the parent
directory of the item and if the item was a directory we invalidate it
too.
2019-10-21 20:51:11 +01:00
Nick Craig-Wood
1fe1a19339 vfs: stop empty dirs disappearing when renamed on bucket based remotes
Before this change when we renamed a directory this cleared the
directory cache for the parent directory too.

If the directory was remaining in the same parent this wasn't
necessary and caused the empty directory to fall out of the cache.

Fixes 
2019-10-14 14:38:30 +01:00