119 Commits

Author	SHA1	Message	Date
Nick Craig-Wood	6fd9e3d717	build: reformat comments to pass go1.19 vet ... See: https://go.dev/doc/go1.19#go-doc	2022-08-05 16:35:41 +01:00
Nick Craig-Wood	a875320e37	sync,operations: optimise --copy-dest and --compare-dest ... Before this change --compare-dest and --copy-dest would check to see if the compare/copy object existed first, before seeing if the destination object was present. This is inefficient, because in most --copy-dest syncs the destination will be present and the compare/copy object need never be tested. --compare-dest syncs may also be speeded up if they are done to the same directory repeatedly. This fixes the problem by re-arranging the logic so if the transfer is not needed then the compare/copy object is never tested. See: https://forum.rclone.org/t/union-with-copy-dest-enabled-is-slower-than-expected/32172	2022-08-03 17:44:47 +01:00
albertony	fdd2f8e6d2	Error strings should not be capitalized ... Reported by staticcheck 2022.1.2 (v0.3.2) See: staticcheck.io	2022-06-23 23:26:02 +02:00
albertony	de5ccaab8e	docs: cross link doc pages for related commands	2022-06-21 23:43:00 +02:00
Nick Craig-Wood	4ac875a811	sync: fix --max-duration and --cutoff-mode soft ... Before this change using --max-duration and --cutoff-mode soft would work like --cutoff-mode hard. This bug was introduced in this commit which made transfers be cancelable - before that transfers couldn't be canceled. 122a47fba6 accounting: Allow transfers to be canceled with context #3257 This change adds the timeout to the input context for reading files rather than the transfer context so the files transfers themselves aren't canceled if --cutoff-mode soft is in action. This also adds a test for cutoff mode soft and max duration which was missing. See: https://forum.rclone.org/t/max-duration-and-retries-not-working-correctly/27738	2022-06-16 22:13:50 +01:00
Nick Craig-Wood	cfe0911e0d	sync: fix tests for overlapping with filter ... In commit 3ccf222acb sync: overlap check is now filter-sensitive The tests were attempting to write invalid objects on some backends due to a leading / on the object name. This fix also adds a few more test cases and makes sure the tests can be run individually.	2022-06-09 14:26:43 +01:00
albertony	f4f0e444bf	filter: allow multiple --exclude-if-present flags - fixes #6219	2022-06-08 17:11:52 +02:00
Nick	3ccf222acb	sync: overlap check is now filter-sensitive ... Previously, the overlap check was based on simple prefix checks of the source and destination paths. Now it actually checks whether the destination is excluded via any filter rule or a "--exclude-if-present"-file.	2022-06-01 18:24:54 +01:00
Nick Craig-Wood	21355b4208	sync: Fix --max-duration so it doesn't retry when the duration is exceeded ... Before this change, if the --max-duration limit was reached then rclone would retry the sync as a fatal error wasn't raised. This checks the deadline and raises a fatal error if necessary at the end of the sync. Fixes #6002	2022-03-03 17:08:16 +00:00
database64128	a7a8372976	🧪 fstest: fix time tests on Windows and add convenience methods to check local and remote fs with precision ... Previously only the fs being checked on gets passed to GetModifyWindow(). However, in most tests, the test files are generated in the local fs and transferred to the remote fs. So the local fs time precision has to be taken into account. This meant that on Windows the time tests failed because the local fs has a time precision of 100ns. Checking remote items uploaded from local fs on Windows also requires a modify window of 100ns.	2021-11-09 11:43:36 +00:00
Nick Craig-Wood	e43b5ce5e5	Remove github.com/pkg/errors and replace with std library version ... This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.	2021-11-07 11:53:30 +00:00
Ivan Andreev	b323bf34e2	sync/test: skip test ConcurrentTruncate on uptobox (take 2) ... The test is not applicable to uptobox which can't upload empty files. The test was not skipped as intended because the direct error was compared. This fix will compare error Cause because Sync wraps the error.	2021-11-02 19:24:23 +03:00
Ivan Andreev	70297c3aed	sync/test: TestConcurrentTruncate needs empty files - skip on uptobox	2021-10-28 17:04:56 +03:00
albertony	e2f47ecdeb	docs: punctuation cleanup ... See #5538	2021-10-20 22:56:19 +02:00
Nathan Collins	c32d5dd1f3	fs: move with --ignore-existing will not delete skipped files - #5463	2021-08-01 17:46:45 +01:00
Nick Craig-Wood	ba5c559fec	fs/sync: fix tests by only --compare-dest timestamp if have hash ... This fixes the integration test errors introduced in #5410	2021-07-07 16:59:51 +01:00
Ivan Andreev	4680c0776d	backend/local: skip entries removed concurrently with List() (#5297) ... This change fixes the bug described below: if a file is removed while the local backend List() runs, the call will flag an accounting error. The bug manifests itself if local backend is the Sync target due to intrinsic concurrency. The odds to hit this bug depend on --checkers and --transfers. Chunker over local backend is affected even more because updating a composite object with a smaller size content translates into removing chunks on the underlying file system and involves a number of List() calls.	2021-07-07 16:50:19 +03:00
Nick Gaya	40024990b7	fs/operations: Don't update timestamps of files in --compare-dest	2021-07-05 16:29:44 +01:00
albertony	98579608ec	docs: cleanup spelling of size and rate units	2021-04-27 02:25:52 +03:00
Nick Craig-Wood	54771e4402	sync: fix incorrect error reported by graceful cutoff - fixes #5203 ... Before this change, a sync which was finished with a graceful transfer cutoff could return "context canceled" instead of the correct error. This fixes the problem by ignoring "context canceled" errors if we have done a graceful stop.	2021-04-06 13:08:42 +01:00
Nick Gaya	9db7c51eaa	sync: don't warn about --no-traverse when --files-from is set	2021-04-05 20:36:39 +01:00
Nick Craig-Wood	25d5ed763c	fs: make sync and operations tests use context instead of global variables	2021-03-15 19:22:07 +00:00
K265	930bca2478	feat: add multiple paths support to --compare-dest and --copy-dest flag	2021-02-16 18:17:04 +00:00
Nick Craig-Wood	f1147fe1dd	rc: sync,copy,move: document createEmptySrcDirs parameter - fixes #4489	2021-02-08 12:25:40 +00:00
Nick Craig-Wood	0a932dc1f2	Add --bwlimit for upload and download #1873	2021-01-30 17:39:30 +00:00
Nick Craig-Wood	3b6df71838	accounting: refactor bwlimit code to allow for multiple slots	2021-01-30 17:39:30 +00:00
Nick Craig-Wood	06f1c0c61c	move: fix data loss when moving the same object ... This checks to see if IDs are the same of the source and destination object before deleting one of them and potentially causing data loss. See: https://forum.rclone.org/t/files-deleted-and-not-moved/21830	2021-01-28 16:14:16 +00:00
Nick Craig-Wood	963c0f28b9	sync: Only print "There was nothing to transfer" if no errors ... See: https://forum.rclone.org/t/immutable-should-set-retries-1-when-source-and-dest-are-different/21326	2020-12-31 18:16:51 +00:00
Nick Craig-Wood	b3815dc0c2	sync: fix --immutable errors retrying many times ... See: https://forum.rclone.org/t/immutable-should-set-retries-1-when-source-and-dest-are-different/21326	2020-12-31 18:16:51 +00:00
Nick Craig-Wood	c22d04aa30	filter: deglobalise to put filter config into the context #4685	2020-11-27 17:28:42 +00:00
Nick Craig-Wood	2e21c58e6a	fs: deglobalise the config #4685 ... This is done by making fs.Config private and attaching it to the context instead. The Config should be obtained with fs.GetConfig and fs.AddConfig should be used to get a new mutable config that can be changed.	2020-11-26 16:40:12 +00:00
Nick Craig-Wood	1fb6ad700f	accounting: add context.Context #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	e3fe31f7cb	fs: add context.Context to fs.GetModifyWindow #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	d846210978	fs: Add context to NewFs #3257 #4685 ... This adds a context.Context parameter to NewFs and related calls. This is necessary as part of reading config from the context - backends need to be able to read the global config.	2020-11-09 18:05:54 +00:00
Josh Soref	e4a87f772f	docs: spelling: e.g. ... Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-28 18:16:23 +00:00
Josh Soref	bbe7eb35f1	docs: spelling: server-side ... Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-28 18:16:23 +00:00
Josh Soref	d0888edc0a	Spelling fixes ... Fix spelling of: above, already, anonymous, associated, authentication, bandwidth, because, between, blocks, calculate, candidates, cautious, changelog, cleaner, clipboard, command, completely, concurrently, considered, constructs, corrupt, current, daemon, dependencies, deprecated, directory, dispatcher, download, eligible, ellipsis, encrypter, endpoint, entrieslist, essentially, existing writers, existing, expires, filesystem, flushing, frequently, hierarchy, however, implementation, implements, inaccurate, individually, insensitive, longer, maximum, metadata, modified, multipart, namedirfirst, nextcloud, obscured, opened, optional, owncloud, pacific, passphrase, password, permanently, persimmon, positive, potato, protocol, quota, receiving, recommends, referring, requires, revisited, satisfied, satisfies, satisfy, semver, serialized, session, storage, strategies, stringlist, successful, supported, surprise, temporarily, temporary, transactions, unneeded, update, uploads, wrapped Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-14 15:21:31 +01:00
Nick Craig-Wood	db4bbf9521	operations: fix use of --suffix without --backup-dir ... As part of the original work adding this feature it was overlooked that this didn't actually work for full rclone copy/sync. This commit fixes the problem and adds a test to make sure it stays working. See: https://forum.rclone.org/t/suffix-not-working-on-folder-upload-via-ssh-sftp/19526	2020-10-04 16:40:20 +01:00
Nick Craig-Wood	75de30cfa8	sync: fix --cutoff-mode soft & cautious so it doesn't end the transfer early ... Before ths fix --cutoff-mode soft and cautious would emit a Fatal error which stopped the sync immediately. This fix introduces a new error which is checked in the sync error processing which stops the sync gracefully. Fixes #4576	2020-09-09 12:53:21 +01:00
Nick Craig-Wood	122a47fba6	accounting: Allow transfers to be canceled with context #3257 ... This makes all transfers cancelable even if the backend doesn't support context as all transfers are done using the Accounting framework.	2020-07-28 16:41:17 +01:00
Nick Craig-Wood	8f42532b6d	sync: add --track-renames-strategy leaf ... See: https://forum.rclone.org/t/how-to-minimize-bandwith-w-r-t-renames-during-sync/16928/22	2020-07-28 11:34:27 +01:00
Nick Craig-Wood	811b30d116	sync: fix deadlock with --track-renames-strategy modtime - fixes #4427 ... Before this change we could exit the popRenameMap function with the lock held. This fixes the problem by defer-ring the unlock. See: https://forum.rclone.org/t/track-renames-strategy-modtime-doesnt-work/16992	2020-07-17 17:09:58 +01:00
Nick Craig-Wood	fb06427c69	sync: fix --track-renames-strategy modtime ... Before this change `--track-renames-strategy` was broken. The hashing method it used could declare times that were very close together to be different. The time hash was discarded and instead we check the modification time window on every hash match. Provided that the user doesn't use `--track-renames-strategy` on a huge number of identically sized files this will perform just fine. See: https://forum.rclone.org/t/track-renames-strategy-modtime-doesnt-work/16992/5	2020-06-12 15:38:35 +01:00
Nick Craig-Wood	b23cf58a41	operations: Add skip all, do all, quit operations to --interactive - fixes #3886 ... This also adds SkipDestructive into all the remaing places --dry-run was used and adds documentation.	2020-06-10 12:33:53 +01:00
Martin Michlmayr	041b201abd	doc: fix typos throughout docs and code	2020-05-25 11:23:58 +01:00
Nick Craig-Wood	518d39815c	sync,copy,move: add --check-first to do all checking before starting transfers ... See: https://forum.rclone.org/t/rclone-sync-doing-transfer-and-checking-in-paralel/16352/	2020-05-22 17:50:07 +01:00
Nick Craig-Wood	147f97d1f7	sync: allow --max-backlog to be -ve meaning as large as possible	2020-05-22 17:50:07 +01:00
Martin Michlmayr	4aee962233	doc: fix typos throughout docs and code	2020-05-20 15:54:51 +01:00
Ben Zenker	899c8e0697	march: added flag to allow Unicode filenames to remain unique ... If your filenames contain two near-identical Unicode characters, rclone will normalize these, making them identical. This flag gives you the ability to keep them unique. This might create unintended side effects, such as duplicating files that contain certain Unicode characters, when downloading them from certain cloud providers to a macOS filesystem. Fixes #4228	2020-05-15 12:28:01 +01:00
Nick Craig-Wood	b2c9ef23fa	sync: make --track-renames tests only check rename count if expecting renames	2020-03-31 10:58:49 +01:00