26 Commits

Author	SHA1	Message	Date
Nick Craig-Wood	27b281ef69	chunkedreader: add --vfs-read-chunk-streams to parallel read chunks ... This converts the ChunkedReader into an interface and provides two implementations one sequential and one parallel. This can be used to improve the performance of the VFS on high bandwidth or high latency links. Fixes #4760	2024-08-14 21:13:09 +01:00
Nick Craig-Wood	a28287e96d	vfs: convert vfs options to new style ... This also - move in use options (Opt) from vfsflags to vfscommon - change os.FileMode to vfscommon.FileMode in parameters - rework vfscommon.FileMode and add tests	2024-07-15 11:09:54 +01:00
Nick Craig-Wood	f3f743c3f9	vfs: fix download loop when file size shrunk ... Before this change, if a file shrunk in size on the remote then rclone could get into an loop trying to download the file forever. The symptom was repeating errors like this: vfs cache: restart download failed: failed to start downloader: failed to open downloader: vfs reader: failed to open source file: invalid seek position The fix was to check that file size in various places and makes sure that we weren't trying to download too much data. This was a problems with backends (like s3) which update the size of the object on Open to the actual size of the object.	2024-03-26 17:32:10 +00:00
Nick Craig-Wood	91b54aafcc	rc: add srcFs and dstFs to core/stats and core/transferred stats ... Before this change it wasn't possible to see where transfers were going from and to in core/stats and core/transferred. When use in rclone mount in particular this made interpreting the stats very hard.	2024-02-02 11:43:10 +00:00
Dimitri Papadopoulos Orfanos	3d473eb54e	docs: fix typos found by codespell in docs and code comments	2023-09-23 12:20:01 +01:00
Eng Zer Jun	8e507075d1	test: replace defer cleanup with t.Cleanup ... Reference: https://pkg.go.dev/testing#T.Cleanup Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	2022-12-09 14:38:05 +00:00
Nick Craig-Wood	617c5d5e1b	rcat: preserve metadata when Copy falls back to Rcat ... Before this change if we copied files of unknown size, then they lost their metadata. This was particularly noticeable using --s3-decompress. This change adds metadata to Rcat and RcatSized and changes Copy to pass the metadata in when it calls Rcat for an unknown sized input. Fixes #6546	2022-11-10 12:04:35 +00:00
albertony	5d6b8141ec	Replace deprecated ioutil ... As of Go 1.16, the same functionality is now provided by package io or package os, and those implementations should be preferred in new code.	2022-11-07 11:41:47 +00:00
albertony	555def2da7	build: add package comments to silence revive linter	2022-08-28 13:43:51 +02:00
Nick Craig-Wood	6fd9e3d717	build: reformat comments to pass go1.19 vet ... See: https://go.dev/doc/go1.19#go-doc	2022-08-05 16:35:41 +01:00
albertony	ec117593f1	Fix lint issues reported by staticcheck ... Used staticcheck 2022.1.2 (v0.3.2) See: staticcheck.io	2022-06-13 21:13:50 +02:00
Nick Craig-Wood	e43b5ce5e5	Remove github.com/pkg/errors and replace with std library version ... This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.	2021-11-07 11:53:30 +00:00
albertony	2925e1384c	Use binary prefixes for size and rate units ... Includes adding support for additional size input suffix Mi and MiB, treated equivalent to M. Extends binary suffix output with letter i, e.g. Ki and Mi. Centralizes creation of bit/byte unit strings.	2021-04-27 02:25:52 +03:00
Nick Craig-Wood	b80d498304	vfs: fix file leaks with --vfs-cache-mode full and --buffer-size 0 ... Before this change using --vfs-cache-mode full and --buffer-size 0 together caused the vfs downloader to open more and more downloaders. This is fixed by introducing a minimum size of 1M for the window to look for an existing downloader. Fixes #4892	2021-01-21 18:35:04 +00:00
Nick Craig-Wood	2e21c58e6a	fs: deglobalise the config #4685 ... This is done by making fs.Config private and attaching it to the context instead. The Config should be obtained with fs.GetConfig and fs.AddConfig should be used to get a new mutable config that can be changed.	2020-11-26 16:40:12 +00:00
Nick Craig-Wood	1fb6ad700f	accounting: add context.Context #3257 #4685	2020-11-09 18:05:54 +00:00
Josh Soref	d0888edc0a	Spelling fixes ... Fix spelling of: above, already, anonymous, associated, authentication, bandwidth, because, between, blocks, calculate, candidates, cautious, changelog, cleaner, clipboard, command, completely, concurrently, considered, constructs, corrupt, current, daemon, dependencies, deprecated, directory, dispatcher, download, eligible, ellipsis, encrypter, endpoint, entrieslist, essentially, existing writers, existing, expires, filesystem, flushing, frequently, hierarchy, however, implementation, implements, inaccurate, individually, insensitive, longer, maximum, metadata, modified, multipart, namedirfirst, nextcloud, obscured, opened, optional, owncloud, pacific, passphrase, password, permanently, persimmon, positive, potato, protocol, quota, receiving, recommends, referring, requires, revisited, satisfied, satisfies, satisfy, semver, serialized, session, storage, strategies, stringlist, successful, supported, surprise, temporarily, temporary, transactions, unneeded, update, uploads, wrapped Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-14 15:21:31 +01:00
Leo Luan	5f1d5a1897	vfs: Fix a deadlock vulnerability in downloaders.Close ... The downloaders.Close() call acquires the downloaders' mutex before calling the wait group wait and the main downloaders thread has a periodical (5 seconds interval) call to kick its waiters and the waiter dispatch function tries to get the mutex. So a deadlock can occur if the Close() call starts, gets the mutex, while the main downloader thread already got the timer's tick and proceeded to call kickWaiters. The deadlock happens when the Close call gets the mutex between the timer's kick and the main downloader thread gets the mutex first. So it's a pretty short period of time and it probably explains why the problem has not surfaced, maybe something like tens of nanoseconds out of 5 seconds (~10^^-8). It took 5 days of continued stressing the Close calls for the deadlock to appear.	2020-10-06 16:35:23 +01:00
Leo Luan	c665201b85	vfs: support synchronous cache space recovery upon ENOSPC ... This patch provides the support of synchronous cache space recovery to allow read threads to recover from ENOSPC errors when cache space can be recovered from cache items that are not in use or safe to be reset/emptied . The patch complements the existing cache cleaning process in two ways. Firstly, the existing cache cleaning process is time-driven that runs periodically. The cache space can run out while the cache cleaner thread is still waiting for its next scheduled run. The io threads encountering ENOSPC return an internal error to the applications in this case even when cache space can be recovered to avoid this error. This patch addresses this problem by having the read threads kick the cache cleaner thread in this condition to recover cache space preventing unnecessary ENOSPC errors from being seen by the applications. Secondly, this patch enhances the cache cleaner to support cache item reset. Currently the cache purge process removes cache items that are not in use. This may not be sufficient when the total size of the working set exceeds the cache directory's capacity. Like in the current code, this patch starts the purge process by removing cache files that are not in use. Cache items whose access times are older than vfs-cache-max-age are removed first. After that, other not-in-use items are removed in LRU order until vfs-cache-max-size is reached. If the vfs-cache-max-size (the quota) is still not reached at this time, this patch adds a cache reset step to reset/empty cache files that are still in use but not dirtied. This enables application processes to continue without seeing an error even when the working set depletes the cache space as long as there is not a large write working set hoarding the entire cache space. By design this patch does not add ENOSPC error recovery for write IOs. Rclone does not empty a write cache item until the file data is written back to the backend upon close. Allowing more cache space to be consumed by dirty cache items when the cache space is already running low would increase the risk of exhausting the cache space in a way that the vfs mount becomes unreadable.	2020-08-25 21:12:06 +01:00
Nick Craig-Wood	4d7f91309b	vfs: fix download threads timing out ... Before this fix, download threads would fill up the buffer and then timeout even though data was still being read from them. If the client was streaming slower than network speed this caused the downloader to stop and be restarted continuously. This caused more potential for skips in the download and unecessary network transactions. This patch fixes that behaviour - as long as a downloader is being read from more often than once every 5 seconds, it won't timeout. This was done by: - kicking the downloader whenever ensureDownloader is called - making the downloader loop if it has already downloaded past the maxOffset - making setRange() always kick the downloader	2020-08-06 17:26:18 +01:00
Nick Craig-Wood	109b695621	vfs: add --vfs-read-ahead parameter for use with --vfs-cache-mode full ... This parameter causes extra read-ahead over --buffer-size which is not buffered in memory but on disk.	2020-08-06 17:26:18 +01:00
Nick Craig-Wood	421585dd72	accounting: add context to Account and propagate changes #3257 ... This is preparation for getting the Accounting to check the context, buf first we need to get it in place. Since this is one of those changes that makes lots of noise, this is in a seperate commit.	2020-07-28 16:41:17 +01:00
Nick Craig-Wood	b2f4f52b64	vfs cache: make logging consistent and remove some debug logging	2020-07-06 17:32:53 +01:00
Nick Craig-Wood	8301a72453	vfs: Fix over downloading with --vfs-cache-mode full and --buffer-size 0 ... This was caused by the signal to stop buffering being ignored when there was no buffer! This is fixed by explicitly checking for no buffering and stopping.	2020-06-30 12:03:39 +01:00
Nick Craig-Wood	143abe39f2	vfs: add tests for downloaders	2020-06-30 12:03:39 +01:00
Nick Craig-Wood	ee04732cbb	vfs: factor writeback and downloaders into their own packages	2020-06-30 12:03:39 +01:00