Before this change if the VFS took more than 5 to initialise (which
can happen if there is a lot of files or a lot of files which need
uploading) the backend was dropped out of the cache before the VFS was
fully created.
This was noticeable in the dropbox backend where the batcher Shutdown
too soon and prevented further uploads.
This fixes the problem by Pinning backends before the VFS cache is
created.
https://forum.rclone.org/t/if-more-than-251-elements-in-the-que-to-upload-fails-with-batcher-is-shutting-down/38076/2
This error happened on a restart of the VFS with files to upload into
a new directory on a bucket based backend. Rclone was assuming that
directories created before the restart would still exist, but this is
a bad assumption for bucket based backends which don't really have
directories.
This change creates the pretend directory and thus the directory cache
if the parent directory does not exist when adding a virtual on a
backend which can't have empty directories.
See: https://forum.rclone.org/t/that-pesky-failed-to-reload-error-message/34527
Includes adding support for additional size input suffix Mi and MiB, treated equivalent to M.
Extends binary suffix output with letter i, e.g. Ki and Mi.
Centralizes creation of bit/byte unit strings.
Some backends, most notably S3, do not report the amount of bytes used.
This patch introduces a new flag that allows instead of relying on the
backend, use recursive scan similar to `rclone size` to compute the total
used space. However, this is ineffective and should be used as a last resort.
Co-authored-by: Yves G <theYinYeti@yalis.fr>
Before this change, if we restarted an upload after a restart then the
file would get uploaded but never added to the directory listings.
This change makes sure we add virtual items to the directory cache
when reloading the cache so that they show up properly.
Before this change we initialized the rc for a single VFS. However
rclone can have multiple VFSes in use now so this is no longer
adequate.
This change adds an optional fs parameter to all the VFS methods to
disambiguate VFSes when there is more than one in use.
It also adds a method vfs/list to show all the active VFSes.
This adds outline tests for the rc commands which didn't have tests
before.
In
54deb01f00 vfs: Make OpenFile and friends return EINVAL if O_RDONLY and O_TRUNC
The generated file open_test.go was edited directly without editing
the generator.
This commit brings the generator make_open_tests.go back into line
with that edit. It also makes it so `go generate` can be used to
regenerate the tests.
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 5ms)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
See: https://forum.rclone.org/t/constantly-high-iowait-add-log/14156
Before this change, change notify polls would clear the directory
cache recursively. So uploading a file to the root would clear the
entire directory cache.
After this change we just invalidate the directory cache of the parent
directory of the item and if the item was a directory we invalidate it
too.
If a file handle is duplicated with dup() and the duplicate handle is
flushed, rclone will go ahead and close the file, making the original
file handle stale. This change removes the close() call from Flush() and
replaces it with FlushWrites() so that the file only gets closed when
Release() is called. The new FlushWrites method takes care of actually
writing the file back to the underlying storage.
Fixes#3381
- Change rclone/fs interfaces to accept context.Context
- Update interface implementations to use context.Context
- Change top level usage to propagate context to lover level functions
Context propagation is needed for stopping transfers and passing other
request-scoped values.
Reduce the number of nodes purged from the dir-cache when ForgetPath is
called. This is done by only forgetting the cache of the received path
and invalidating the parent folder cache by resetting *Dir.read.
The parent will read the listing on the next access and reuse the
dir-cache of entries in *Dir.items.
This stops the cache cleaner running unnecessarily and saves
resources.
This also helps with issue #2227 which was caused by a second mount
deleting objects in the first mounts cache.
