62 Commits

Author	SHA1	Message	Date
Nick Craig-Wood	6001f05a12	union: the root folder shows the modtime configured by --default-time #6986	2023-05-04 15:03:11 +01:00
albertony	9cea493f58	union: remove unused code (fixes issue reported by the unused linter)	2023-03-26 14:28:15 +02:00
albertony	5d6b8141ec	Replace deprecated ioutil ... As of Go 1.16, the same functionality is now provided by package io or package os, and those implementations should be preferred in new code.	2022-11-07 11:41:47 +00:00
Lesmiscore	6654b66114	union: propagate SlowHash feature	2022-10-10 07:58:01 +01:00
albertony	555def2da7	build: add package comments to silence revive linter	2022-08-28 13:43:51 +02:00
Nick Craig-Wood	6fd9e3d717	build: reformat comments to pass go1.19 vet ... See: https://go.dev/doc/go1.19#go-doc	2022-08-05 16:35:41 +01:00
r-ricci	67fd60275a	union: fix panic due to misalignment of struct field in 32 bit architectures ... `FS.cacheExpiry` is accessed through sync/atomic. According to the documentation, "On ARM, 386, and 32-bit MIPS, it is the caller's responsibility to arrange for 64-bit alignment of 64-bit words accessed atomically. The first word in a variable or in an allocated struct, array, or slice can be relied upon to be 64-bit aligned." Before commit 1d2fe0d856 this field was aligned, but then a new field was added to the structure, causing the test suite to panic on linux/386. No other field is used with sync/atomic, so `cacheExpiry` can just be placed at the beginning of the stuct to ensure it is always aligned.	2022-07-11 18:34:06 +01:00
Nick Craig-Wood	b310490fa5	union: fix multiple files being uploaded when roots don't exist ... See: https://forum.rclone.org/t/union-backend-copying-to-all-remotes-while-it-shouldnt/31781	2022-07-11 18:19:36 +01:00
Nick Craig-Wood	0ee0812a2b	union: fix duplicated files when using directories with leading / ... See: https://forum.rclone.org/t/union-backend-copying-to-all-remotes-while-it-shouldnt/31781	2022-07-11 18:19:36 +01:00
albertony	3435bf7f34	staticcheck: no value of type int64 is greater than math.MaxInt64	2022-07-04 11:24:59 +02:00
albertony	0772cae314	staticcheck: use result of type assertion to simplify cases	2022-07-04 11:24:59 +02:00
Nick Craig-Wood	ed87ae51c0	union: support metadata	2022-06-29 17:30:37 +01:00
Nick Craig-Wood	866c873daa	backend: allow wrapping backend tests to run in make quicktest	2022-06-29 17:30:37 +01:00
Nick Craig-Wood	461d041c4d	fstest: remove spurious contents return from PutTestContents and friends	2022-06-29 11:18:02 +01:00
Nick Craig-Wood	1651429041	union: add min_free_space option for lfs/eplfs policies - fixes #6071	2022-06-16 22:13:50 +01:00
Nick Craig-Wood	29e37749b3	union: fix eplus policy to select correct entry for existing files #6071	2022-06-16 22:13:50 +01:00
Nick Craig-Wood	1e1af46a12	union: fix get free space for remotes which don't support it #6071 ... Before this fix GetFreeSpace returned math.MaxInt64 for remotes which don't support reading free space, however this is used in various comparison routines as a too large value, meaning that remotes of size math.MaxInt64 were never being selected. This fixes GetFreeSpace to return math.MaxInt64 - 1 so then can be selected. It also fixes GetUsedSpace the same way however as the default for not supported was 0 this was very unlikely to have ever caused a problem.	2022-06-16 22:13:50 +01:00
Nick Craig-Wood	1d2fe0d856	union: enable passing of options to upstreams and policies #6071 ... This factors out the options into a sub package so they can be passed to upstreams and used in policies.	2022-06-16 22:13:50 +01:00
Nick Craig-Wood	c2baacc0a4	union: fix uploading files to union of all bucket based remotes ... Before this fix, if uploading to a union consisting of all bucket based remotes (eg s3), uploads failed with: Failed to copy: object not found This was because the union backend was relying on parent directories being created to work out which files to upload. If all the upstreams were bucket based backends which can't hold empty directories, no directories were created and the upload failed. This fixes the problem by returning the upstreams used when creating the directory for the upload, rather than searching for them again after they've been created. This will also make the union backend a little more efficient. Fixes #6170	2022-05-19 13:23:41 +01:00
Nick Craig-Wood	5dd51e6149	union: fix deadlock when one part of a multi-upload fails ... Before this fix, rclone would deadlock when uploading two files at once, if one errored. This caused the other file to block in the multi reader and never complete. This fix drains the input buffer on error which allows the other upload to complete. See: https://forum.rclone.org/t/union-with-create-policy-all-copy-stuck-when-first-union-fails/29601	2022-03-09 11:30:55 +00:00
Eng Zer Jun	8cf76f5e11	test: use T.TempDir to create temporary test directory ... The directory created by `T.TempDir` is automatically removed when the test and all its subtests complete. Reference: https://pkg.go.dev/testing#T.TempDir Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	2022-02-01 11:47:04 +00:00
albertony	bd4bbed592	config: remove explicit setting of required property to true for options with a default value ... See #5551	2022-01-16 14:57:38 +01:00
Nick Craig-Wood	f26abc89a6	union: fix treatment of remotes with // in ... See: https://forum.rclone.org/t/connection-string-with-union-backend-and-a-lot-of-quotes/27577	2021-11-23 17:41:12 +00:00
Nick Craig-Wood	96e099d8e7	union: fix error handling broken by removal of github.com/pkg/errors ... There were instances of errors.Wrap being called with a nil error which the conversion didn't deal with correctly.	2021-11-09 13:43:45 +00:00
Vitor Arruda	9beb0677e4	backend: Fix union eplus policy returned nil	2021-11-08 11:55:27 +00:00
Nick Craig-Wood	e43b5ce5e5	Remove github.com/pkg/errors and replace with std library version ... This is possible now that we no longer support go1.12 and brings rclone into line with standard practices in the Go world. This also removes errors.New and errors.Errorf from lib/errors and prefers the stdlib errors package over lib/errors.	2021-11-07 11:53:30 +00:00
albertony	e2f47ecdeb	docs: punctuation cleanup ... See #5538	2021-10-20 22:56:19 +02:00
Nick Craig-Wood	bb0c4ad2d8	union: fix rename not working with union of local disk and bucket based remote ... Before this change the union's feature flags were a strict AND of the underlying remotes. This means that a union of a local disk (which can Move but not Copy) and a bucket based remote (which can Copy but not Move) could neither Move nor Copy. This fix advertises Move in the union if all the remotes can Move or Copy. It also implements Move as Copy+Delete (like rclone does normally) if the underlying union does not support Move. This enables renames to work with unions of local disk and bucket based remotes expected. Fixes #5632	2021-09-30 20:09:02 +01:00
Nick Craig-Wood	b8c1cf7451	union: fix initialisation broken in refactor - fixes #5139 ... This commit broke the initialisation of the union backend f17d7c0012 union: refactor to use fspath.SplitFs instead of fs.ParseRemote #4996 This patch fixes it.	2021-03-24 09:47:38 +00:00
Nick Craig-Wood	f17d7c0012	union: refactor to use fspath.SplitFs instead of fs.ParseRemote #4996	2021-03-15 19:22:07 +00:00
Nick Craig-Wood	ebd9462ea6	union: fix union attempting to update files on a read only file system ... Before this change, when using an all create method with one of the upstreams being read only, if there was an existing file on the read only remote, it was impossible to update it. This change detects that situation and creates the file on a read/write upstream. This file will shadow the file on the read/only upstream. If it is deleted the read only upstream file will be visible again. Fixes #4929	2021-03-15 19:22:07 +00:00
Nick Craig-Wood	6b9e4f939d	union: fix crash when using epff policy - fixes #5000 ... Before this fix using the epff policy could double close a channel. The fix refactors the code to make that impossible and cancels any running queries when the first query is found.	2021-03-15 19:22:07 +00:00
Nick Craig-Wood	d809930e1d	union: fix mkdir at root with remote:/ ... Before the this fix if you specified remote:/ then the union backend would fail to notice the root directory existed. This was fixed by stripping the trailing / from the root. See: https://forum.rclone.org/t/upgraded-from-1-45-to-1-54-now-cant-create-new-directory-within-union-mount/22284/	2021-02-17 12:11:34 +00:00
Nick Craig-Wood	47aada16a0	fs: add Shutdown optional method for backends	2020-11-27 17:35:01 +00:00
Nick Craig-Wood	f0905499e3	random: seed math/rand in one place with crypto strong seed #4783 ... This shouldn't be read as encouraging the use of math/rand instead of crypto/rand in security sensitive contexts, rather as a safer default if that does happen by accident.	2020-11-18 17:48:44 +00:00
Nick Craig-Wood	8b96933e58	fs: Add context to fs.Features.Fill & fs.Features.Mask #3257 #4685	2020-11-09 18:05:54 +00:00
Nick Craig-Wood	d846210978	fs: Add context to NewFs #3257 #4685 ... This adds a context.Context parameter to NewFs and related calls. This is necessary as part of reading config from the context - backends need to be able to read the global config.	2020-11-09 18:05:54 +00:00
Josh Soref	bbe7eb35f1	docs: spelling: server-side ... Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-28 18:16:23 +00:00
Nick Craig-Wood	cf0bdad5de	union: create root directories if none exist ... This fixes the TestUnion: integration test if the /tmp/union[123] dirs don't exist.	2020-10-25 18:10:49 +00:00
Josh Soref	d0888edc0a	Spelling fixes ... Fix spelling of: above, already, anonymous, associated, authentication, bandwidth, because, between, blocks, calculate, candidates, cautious, changelog, cleaner, clipboard, command, completely, concurrently, considered, constructs, corrupt, current, daemon, dependencies, deprecated, directory, dispatcher, download, eligible, ellipsis, encrypter, endpoint, entrieslist, essentially, existing writers, existing, expires, filesystem, flushing, frequently, hierarchy, however, implementation, implements, inaccurate, individually, insensitive, longer, maximum, metadata, modified, multipart, namedirfirst, nextcloud, obscured, opened, optional, owncloud, pacific, passphrase, password, permanently, persimmon, positive, potato, protocol, quota, receiving, recommends, referring, requires, revisited, satisfied, satisfies, satisfy, semver, serialized, session, storage, strategies, stringlist, successful, supported, surprise, temporarily, temporary, transactions, unneeded, update, uploads, wrapped Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>	2020-10-14 15:21:31 +01:00
Nick Craig-Wood	70c8566cb8	fs: Pin created backends until parents are finalized ... This attempts to solve the backend lifecycle problem by - Pinning backends mentioned on the command line into the cache indefinitely - Unpinning backends when the containing structure (VFS, wrapping backend) is destroyed See: https://forum.rclone.org/t/rclone-rc-backend-command-not-working-as-expected/18834	2020-09-01 18:21:03 +01:00
Nick Craig-Wood	23c826db52	union: fix writing with the all policy - fixes #4534 ... Before this change writing with the all policy deadlocked while uploading. This change fixes the problem by fixing the multi reader, closing the pipes at the correct time with the correct error. This is factored into a new function as it was used twice. This patch also adds a new test which tests the all policies.	2020-09-01 18:21:03 +01:00
Nick Craig-Wood	85f9bd1abf	union: fix tests by looking for fs.ErrorDirNotFound found in Purge and About ... Before this change we errored out if one upstream errored in Purge or About. This change checks for fs.ErrorDirNotFound and skips that backend in this case.	2020-08-19 18:04:16 +01:00
Nick Craig-Wood	a2afa9aadd	fs: Add directory to optional Purge interface - fixes #1891 ... - add a directory to the optional Purge interface - fix up all the backends - add an additional integration test to test for the feature - use the new feature in operations.Purge Many of the backends had been prepared in advance for this so the change was trivial for them.	2020-07-31 17:43:17 +01:00
Martin Michlmayr	041b201abd	doc: fix typos throughout docs and code	2020-05-25 11:23:58 +01:00
Martin Michlmayr	4aee962233	doc: fix typos throughout docs and code	2020-05-20 15:54:51 +01:00
Max Sum	33d9310c49	union: enable ListR when upstreams contain local ... Enable fast list functions for union backend when: - at least one of the upstreams supports fast list - upstreams only consist of backends that support fast list and local backend. Fixes #3000	2020-05-13 13:10:35 +01:00
Max Sum	54b16bd054	union: implement ListR	2020-05-10 17:57:03 +00:00
Max Sum	f21e97001b	union: fix server-side copy	2020-05-10 17:56:18 +00:00
Max Sum	78a9e7440a	union: Implement multiple writable remotes	2020-03-21 18:11:24 +00:00