This changes hasVirtual to an atomic struct variable that's updated on
add or delete from the virtual map.
This keeps it up to date and avoids deadlocks.
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
This empties the directory cache after twice the directory cache
period to release memory.
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
Before this change if the VFS took more than 5 to initialise (which
can happen if there is a lot of files or a lot of files which need
uploading) the backend was dropped out of the cache before the VFS was
fully created.
This was noticeable in the dropbox backend where the batcher Shutdown
too soon and prevented further uploads.
This fixes the problem by Pinning backends before the VFS cache is
created.
https://forum.rclone.org/t/if-more-than-251-elements-in-the-que-to-upload-fails-with-batcher-is-shutting-down/38076/2
Before this fix, when a write to a read only directory failed, rclone
would leav spurious directory entries in the directory.
This confuses `rclone serve webdav` into giving this error
http: superfluous response.WriteHeader
This fixes the VFS layer to remove any directory entries where the
file creation did not succeed.
Fixes#5702
This error happened on a restart of the VFS with files to upload into
a new directory on a bucket based backend. Rclone was assuming that
directories created before the restart would still exist, but this is
a bad assumption for bucket based backends which don't really have
directories.
This change creates the pretend directory and thus the directory cache
if the parent directory does not exist when adding a virtual on a
backend which can't have empty directories.
See: https://forum.rclone.org/t/that-pesky-failed-to-reload-error-message/34527
Before this change if a file was uploaded to a backend which didn't
support modtimes, the time of the file read after the upload had
completed would change to the time the file was uploaded on the
backend.
When using `--vfs-cache-mode writes` or `full` this time would be
different by the `--vfs-write-back` delay which would cause
applications to think the file had been modified.
This changes uses the last modification time read by the OS as a
virtual modtime for backends which don't support setting modtimes. It
does not change the modtime to that actually uploaded.
This means that as long as the file remains in the directory cache it
will have the expected modtime.
See: https://forum.rclone.org/t/saving-files-causes-wrong-modified-time-to-be-set-for-a-few-seconds-on-webdav-mount-with-bitrix24/36451
Before this change, if you renamed a directory containg files yet to
be uploaded then deleted the directory the files would still be
uploaded.
This fixes the problem by changing the directory path in all the file
objects in a directory when it is renamed. This wasn't necessary until
we introduced virtual files and directories which lived beyond the
directory flush mechanism.
Fixes#6809
Before this fix, opening a file with `O_CREATE|O_RDONLY` caused an IO error to
be returned when using `--vfs-cache-mode off` or `--vfs-cache-mode writes`.
This was because the file was opened with read intent, but the `O_CREATE`
implies write intent to create the file even though the file is opened
`O_RDONLY`.
This fix sets write intent for the file if `O_CREATE` is passed in which fixes
the problem for all the VFS cache modes.
It also extends the exhaustive open flags testing to `--vfs-cache-mode writes`
as well as `--vfs-cache-mode full` which would have caught this problem.
See: https://forum.rclone.org/t/i-o-error-trashing-file-on-sftp-mount/34317/
Before this patch a deadlock could occur if the cache cleaner was
running when an object upload finished.
This fixes the problem by delaying marking the object as clean until
we have notified the VFS layer. This means that the cache cleaner
won't consider the object until **after** the VFS layer has been
notified, thus avoiding the deadlock.
See: https://forum.rclone.org/t/rclone-mount-deadlock-when-dir-cache-time-strikes/33486/
Before this change we truncated files in the backing store regardless
of whether we needed to or not.
After, we check to see if the file is the right size and don't
truncate if it is.
Apparently Windows Defender likes to check executables each time they
are modified, and truncating a file to its existing size is enough to
trigger the Windows Defender scan. This was causing a big slowdown for
operations which opened and closed the file a lot, such as looking at
properties on an executable.
See: https://forum.rclone.org/t/for-mount-sftp-why-right-click-on-exe-file-is-so-slow-until-it-freezes/33830
Before this change if we copied files of unknown size, then they lost
their metadata.
This was particularly noticeable using --s3-decompress.
This change adds metadata to Rcat and RcatSized and changes Copy to
pass the metadata in when it calls Rcat for an unknown sized input.
Fixes#6546
Before this change the VFS cache cleaner would loop indefinitely while
the cache was above quota. This used up all the CPU.
This fix prevents the cache cleaner from looping. It will be kicked on
ENOSPACE and run in its scheduled time otherwise so this should be
sufficient.
See: https://forum.rclone.org/t/vfs-keeps-checking-same-files/32120
This message is a double panic and was actually caused by an assertion
panic in:
vfs/vfscache/downloaders/downloaders.go
This is triggered by the code added relatively recently to fix a bug
with renaming files:
ec72432cec vfs: fix failed to _ensure cache internal error: downloaders is nil error
So it appears that item.o may be nil at this point.
This patch detects item.o being nil and fetches it again with NewObject.
Fixes#6190Fixes#6235
Before this change we ran the tests and the mount in the same process.
This could cause deadlocks and often did, and made the mount tests
very unreliable.
This fixes the problem by running the mount in a seperate process and
commanding it via a pipe over stdin/stdout.
When using filepath.Dir, a difference to path.Dir is that it returns os PathSeparator
instead of slash when the path consists entirely of separators.
Also fixed casing of the function name, use OS in all caps instead of Os
as recommended here: https://github.com/golang/go/wiki/CodeReviewComments#initialisms
In this commit
f4c40bf79d mount: add --devname to set the device name sent to FUSE for mount display
The --devname parameter was added. However it was soon noticed that
attempting to mount via the rc gave this error:
mount helper error: fusermount: unknown option 'fsname'
mount FAILED: fusermount: exit status 1
This was because the DeviceName (and VolumeName) parameter was never
being initialised when the mount was called via the rc.
The fix for this was to refactor the rc interface so it called the
same Mount method as the command line mount which initialised the
DeviceName and VolumeName parameters properly.
This also fixes the cmd/mount tests which were breaking in the same
way but since they aren't normally run on the CI we didn't notice.
Fixes#6044
This error was caused by renaming an open file.
When the file was renamed in the cache, the downloaders were cleared,
however the downloaders were not re-opened when needed again, instead
this error was generated.
This fix re-opens the downloaders if they have been closed by renaming
the file.
Fixes#5984
Previously only the fs being checked on gets passed to
GetModifyWindow(). However, in most tests, the test files are
generated in the local fs and transferred to the remote fs. So the
local fs time precision has to be taken into account.
This meant that on Windows the time tests failed because the
local fs has a time precision of 100ns. Checking remote items uploaded
from local fs on Windows also requires a modify window of 100ns.
This is possible now that we no longer support go1.12 and brings
rclone into line with standard practices in the Go world.
This also removes errors.New and errors.Errorf from lib/errors and
prefers the stdlib errors package over lib/errors.
