10 Commits

Author	SHA1	Message	Date
nielash	2bebbfaded	bisync: add to integration tests - fixes #7665 ... This change officially adds bisync to the nightly integration tests for all backends. This will be part of giving us the confidence to take bisync out of beta. A number of fixes have been added to account for features which can differ on different backends -- for example, hash types / modtime support, empty directories, unicode normalization, and unimportant differences in log output. We will likely find that more of these are needed once we start running these with the full set of remotes. Additionally, bisync's extremely sensitive tests revealed a few bugs in other backends that weren't previously covered by other tests. Fixes for those issues have been submitted on the following separate PRs (and bisync test failures will be expected until they are merged): - #7670 memory: fix deadlock in operations.Purge - #7688 memory: fix incorrect list entries when rooted at subdirectory - #7690 memory: fix dst mutating src after server-side copy - #7692 dropbox: fix chunked uploads when size <= chunkSize Relatedly, workarounds have been put in place for the following backend limitations that are unsolvable for the time being: - #3262 drive is sometimes aware of trashed files/folders when it shouldn't be - #6199 dropbox can't handle emojis and certain other characters - #4590 onedrive API has longstanding bug for conflictBehavior=replace in server-side copy/move	2024-03-27 10:50:14 -04:00
nielash	810644e873	bisync: add --resync-mode for customizing --resync - fixes #5681 ... Before this change, the path1 version of a file always prevailed during --resync, and many users requested options to automatically select the winner based on characteristics such as newer, older, larger, and smaller. This change adds support for such options. Note that ideally this feature would have been implemented by allowing the existing `--resync` flag to optionally accept string values such as `--resync newer`. However, this would have been a breaking change, as the existing flag is a `bool` and it does not seem to be possible to have a `string` flag that accepts both `--resync newer` and `--resync` (with no argument.) (`NoOptDefVal` does not work for this, as it would force an `=` like `--resync=newer`.) So instead, the best compromise to avoid a breaking change was to add a new `--resync-mode CHOICE` flag that implies `--resync`, while maintaining the existing behavior of `--resync` (which implies `--resync-mode path1`. i.e. both flags are now valid, and either can be used without the other. --resync-mode CHOICE In the event that a file differs on both sides during a `--resync`, `--resync-mode` controls which version will overwrite the other. The supported options are similar to `--conflict-resolve`. For all of the following options, the version that is kept is referred to as the "winner", and the version that is overwritten (deleted) is referred to as the "loser". The options are named after the "winner": - `path1` - (the default) - the version from Path1 is unconditionally considered the winner (regardless of `modtime` and `size`, if any). This can be useful if one side is more trusted or up-to-date than the other, at the time of the `--resync`. - `path2` - same as `path1`, except the path2 version is considered the winner. - `newer` - the newer file (by `modtime`) is considered the winner, regardless of which side it came from. This may result in having a mix of some winners from Path1, and some winners from Path2. (The implementation is analagous to running `rclone copy --update` in both directions.) - `older` - same as `newer`, except the older file is considered the winner, and the newer file is considered the loser. - `larger` - the larger file (by `size`) is considered the winner (regardless of `modtime`, if any). This can be a useful option for remotes without `modtime` support, or with the kinds of files (such as logs) that tend to grow but not shrink, over time. - `smaller` - the smaller file (by `size`) is considered the winner (regardless of `modtime`, if any). For all of the above options, note the following: - If either of the underlying remotes lacks support for the chosen method, it will be ignored and will fall back to the default of `path1`. (For example, if `--resync-mode newer` is set, but one of the paths uses a remote that doesn't support `modtime`.) - If a winner can't be determined because the chosen method's attribute is missing or equal, it will be ignored, and bisync will instead try to determine whether the files differ by looking at the other `--compare` methods in effect. (For example, if `--resync-mode newer` is set, but the Path1 and Path2 modtimes are identical, bisync will compare the sizes.) If bisync concludes that they differ, preference is given to whichever is the "source" at that moment. (In practice, this gives a slight advantage to Path2, as the 2to1 copy comes before the 1to2 copy.) If the files _do not_ differ, nothing is copied (as both sides are already correct). - These options apply only to files that exist on both sides (with the same name and relative path). Files that exist *only* on one side and not the other are *always* copied to the other, during `--resync` (this is one of the main differences between resync and non-resync runs.). - `--conflict-resolve`, `--conflict-loser`, and `--conflict-suffix` do not apply during `--resync`, and unlike these flags, nothing is renamed during `--resync`. When a file differs on both sides during `--resync`, one version always overwrites the other (much like in `rclone copy`.) (Consider using `--backup-dir` to retain a backup of the losing version.) - Unlike for `--conflict-resolve`, `--resync-mode none` is not a valid option (or rather, it will be interpreted as "no resync", unless `--resync` has also been specified, in which case it will be ignored.) - Winners and losers are decided at the individual file-level only (there is not currently an option to pick an entire winning directory atomically, although the `path1` and `path2` options typically produce a similar result.) - To maintain backward-compatibility, the `--resync` flag implies `--resync-mode path1` unless a different `--resync-mode` is explicitly specified. Similarly, all `--resync-mode` options (except `none`) imply `--resync`, so it is not necessary to use both the `--resync` and `--resync-mode` flags simultaneously -- either one is sufficient without the other.	2024-01-20 17:17:01 -05:00
nielash	68f0998699	bisync: add options to auto-resolve conflicts - fixes #7471 ... Before this change, when a file was new/changed on both paths (relative to the prior sync), and the versions on each side were not identical, bisync would keep both versions, renaming them with ..path1 and ..path2 suffixes, respectively. Many users have requested more control over how bisync handles such conflicts -- including an option to automatically select one version as the "winner" and rename or delete the "loser". This change introduces support for such options. --conflict-resolve CHOICE In bisync, a "conflict" is a file that is *new* or *changed* on *both sides* (relative to the prior run) AND is *not currently identical* on both sides. `--conflict-resolve` controls how bisync handles such a scenario. The currently supported options are: - `none` - (the default) - do not attempt to pick a winner, keep and rename both files according to `--conflict-loser` and `--conflict-suffix` settings. For example, with the default settings, `file.txt` on Path1 is renamed `file.txt.conflict1` and `file.txt` on Path2 is renamed `file.txt.conflict2`. Both are copied to the opposite path during the run, so both sides end up with a copy of both files. (As `none` is the default, it is not necessary to specify `--conflict-resolve none` -- you can just omit the flag.) - `newer` - the newer file (by `modtime`) is considered the winner and is copied without renaming. The older file (the "loser") is handled according to `--conflict-loser` and `--conflict-suffix` settings (either renamed or deleted.) For example, if `file.txt` on Path1 is newer than `file.txt` on Path2, the result on both sides (with other default settings) will be `file.txt` (winner from Path1) and `file.txt.conflict1` (loser from Path2). - `older` - same as `newer`, except the older file is considered the winner, and the newer file is considered the loser. - `larger` - the larger file (by `size`) is considered the winner (regardless of `modtime`, if any). - `smaller` - the smaller file (by `size`) is considered the winner (regardless of `modtime`, if any). - `path1` - the version from Path1 is unconditionally considered the winner (regardless of `modtime` and `size`, if any). This can be useful if one side is usually more trusted or up-to-date than the other. - `path2` - same as `path1`, except the path2 version is considered the winner. For all of the above options, note the following: - If either of the underlying remotes lacks support for the chosen method, it will be ignored and fall back to `none`. (For example, if `--conflict-resolve newer` is set, but one of the paths uses a remote that doesn't support `modtime`.) - If a winner can't be determined because the chosen method's attribute is missing or equal, it will be ignored and fall back to `none`. (For example, if `--conflict-resolve newer` is set, but the Path1 and Path2 modtimes are identical, even if the sizes may differ.) - If the file's content is currently identical on both sides, it is not considered a "conflict", even if new or changed on both sides since the prior sync. (For example, if you made a change on one side and then synced it to the other side by other means.) Therefore, none of the conflict resolution flags apply in this scenario. - The conflict resolution flags do not apply during a `--resync`, as there is no "prior run" to speak of (but see `--resync-mode` for similar options.) --conflict-loser CHOICE `--conflict-loser` determines what happens to the "loser" of a sync conflict (when `--conflict-resolve` determines a winner) or to both files (when there is no winner.) The currently supported options are: - `num` - (the default) - auto-number the conflicts by automatically appending the next available number to the `--conflict-suffix`, in chronological order. For example, with the default settings, the first conflict for `file.txt` will be renamed `file.txt.conflict1`. If `file.txt.conflict1` already exists, `file.txt.conflict2` will be used instead (etc., up to a maximum of 9223372036854775807 conflicts.) - `pathname` - rename the conflicts according to which side they came from, which was the default behavior prior to `v1.66`. For example, with `--conflict-suffix path`, `file.txt` from Path1 will be renamed `file.txt.path1`, and `file.txt` from Path2 will be renamed `file.txt.path2`. If two non-identical suffixes are provided (ex. `--conflict-suffix cloud,local`), the trailing digit is omitted. Importantly, note that with `pathname`, there is no auto-numbering beyond `2`, so if `file.txt.path2` somehow already exists, it will be overwritten. Using a dynamic date variable in your `--conflict-suffix` (see below) is one possible way to avoid this. Note also that conflicts-of-conflicts are possible, if the original conflict is not manually resolved -- for example, if for some reason you edited `file.txt.path1` on both sides, and those edits were different, the result would be `file.txt.path1.path1` and `file.txt.path1.path2` (in addition to `file.txt.path2`.) - `delete` - keep the winner only and delete the loser, instead of renaming it. If a winner cannot be determined (see `--conflict-resolve` for details on how this could happen), `delete` is ignored and the default `num` is used instead (i.e. both versions are kept and renamed, and neither is deleted.) `delete` is inherently the most destructive option, so use it only with care. For all of the above options, note that if a winner cannot be determined (see `--conflict-resolve` for details on how this could happen), or if `--conflict-resolve` is not in use, *both* files will be renamed. --conflict-suffix STRING[,STRING] `--conflict-suffix` controls the suffix that is appended when bisync renames a `--conflict-loser` (default: `conflict`). `--conflict-suffix` will accept either one string or two comma-separated strings to assign different suffixes to Path1 vs. Path2. This may be helpful later in identifying the source of the conflict. (For example, `--conflict-suffix dropboxconflict,laptopconflict`) With `--conflict-loser num`, a number is always appended to the suffix. With `--conflict-loser pathname`, a number is appended only when one suffix is specified (or when two identical suffixes are specified.) i.e. with `--conflict-loser pathname`, all of the following would produce exactly the same result: ``` --conflict-suffix path --conflict-suffix path,path --conflict-suffix path1,path2 ``` Suffixes may be as short as 1 character. By default, the suffix is appended after any other extensions (ex. `file.jpg.conflict1`), however, this can be changed with the `--suffix-keep-extension` flag (i.e. to instead result in `file.conflict1.jpg`). `--conflict-suffix` supports several *dynamic date variables* when enclosed in curly braces as globs. This can be helpful to track the date and/or time that each conflict was handled by bisync. For example: ``` --conflict-suffix {DateOnly}-conflict // result: myfile.txt.2006-01-02-conflict1 ``` All of the formats described [here](https://pkg.go.dev/time#pkg-constants) and [here](https://pkg.go.dev/time#example-Time.Format) are supported, but take care to ensure that your chosen format does not use any characters that are illegal on your remotes (for example, macOS does not allow colons in filenames, and slashes are also best avoided as they are often interpreted as directory separators.) To address this particular issue, an additional `{MacFriendlyTime}` (or just `{mac}`) option is supported, which results in `2006-01-02 0304PM`. Note that `--conflict-suffix` is entirely separate from rclone's main `--sufix` flag. This is intentional, as users may wish to use both flags simultaneously, if also using `--backup-dir`. Finally, note that the default in bisync prior to `v1.66` was to rename conflicts with `..path1` and `..path2` (with two periods, and `path` instead of `conflict`.) Bisync now defaults to a single dot instead of a double dot, but additional dots can be added by including them in the specified suffix string. For example, for behavior equivalent to the previous default, use: ``` [--conflict-resolve none] --conflict-loser pathname --conflict-suffix .path ```	2024-01-20 17:17:01 -05:00
nielash	b4216648e4	bisync: full support for comparing checksum, size, modtime - fixes #5679 fixes #5683 fixes #5684 fixes #5675 ... Before this change, bisync could only detect changes based on modtime, and would refuse to run if either path lacked modtime support. This made bisync unavailable for many of rclone's backends. Additionally, bisync did not account for the Fs's precision when comparing modtimes, meaning that they could only be reliably compared within the same side -- not against the opposite side. Size and checksum (even when available) were ignored completely for deltas. After this change, bisync now fully supports comparing based on any combination of size, modtime, and checksum, lifting the prior restriction on backends without modtime support. The comparison logic considers the backend's precision, hash types, and other features as appropriate. The comparison features optionally use a new --compare flag (which takes any combination of size,modtime,checksum) and even supports some combinations not otherwise supported in `sync` (like comparing all three at the same time.) By default (without the --compare flag), bisync inherits the same comparison options as `sync` (that is: size and modtime by default, unless modified with flags such as --checksum or --size-only.) If the --compare flag is set, it will override these defaults. If --compare includes checksum and both remotes support checksums but have no hash types in common with each other, checksums will be considered only for comparisons within the same side (to determine what has changed since the prior sync), but not for comparisons against the opposite side. If one side supports checksums and the other does not, checksums will only be considered on the side that supports them. When comparing with checksum and/or size without modtime, bisync cannot determine whether a file is newer or older -- only whether it is changed or unchanged. (If it is changed on both sides, bisync still does the standard equality-check to avoid declaring a sync conflict unless it absolutely has to.) Also included are some new flags to customize the checksum comparison behavior on backends where hashes are slow or unavailable. --no-slow-hash and --slow-hash-sync-only allow selectively ignoring checksums on backends such as local where they are slow. --download-hash allows computing them by downloading when (and only when) they're otherwise not available. Of course, this option probably won't be practical with large files, but may be a good option for syncing small-but-important files with maximum accuracy (for example, a source code repo on a crypt remote.) An additional advantage over methods like cryptcheck is that the original file is not required for comparison (for example, --download-hash can be used to bisync two different crypt remotes with different passwords.) Additionally, all of the above are now considered during the final --check-sync for much-improved accuracy (before this change, it only compared filenames!) Many other details are explained in the included docs.	2024-01-20 16:08:06 -05:00
nielash	57624629d6	bisync: account for differences in backend features on integration tests - see #5679 ... Before this change, integration tests often could not be run on backends with differing features from the local system that goldenized them. In particular, differences in modtime precision, checksum support, and encoding would cause false positives. After this change, the tests more accurately account for the features of the backend being tested, which allows us to see true positives more clearly, and more meaningfully assess whether a backend is supported.	2024-01-20 14:50:08 -05:00
nielash	9c96c13a35	bisync: optimize --resync performance -- partially addresses #5681 ... Before this change, --resync was handled in three steps, and needed to do a lot of unnecessary work to implement its own --ignore-existing logic, which also caused problems with unicode normalization, in addition to being pretty slow. After this change, it is refactored to produce the same result much more efficiently, by reducing the three steps to two and letting ci.IgnoreExisting do the work instead of reinventing the wheel. The behavior and sync order remain unchanged for now -- just faster (but see the ongoing lively discussions about potential future changes in #5681!)	2024-01-20 14:50:08 -05:00
nielash	fd95511091	bisync: generate listings concurrently with march -- fixes #7332 ... Before this change, bisync needed to build a full listing for Path1, then a full listing for Path2, then compare them -- and each of those tasks needed to finish before the next one could start. In addition to being slow and inefficient, it also caused real problems if a file changed between the time bisync checked it on Path1 and the time it checked the corresponding file on Path2. This change solves these problems by listing both paths concurrently, using the same March infrastructure that check and sync use to traverse two directories in lock-step, optimized by Go's robust concurrency support. Listings should now be much faster, and any given path is now checked nearly-instantaneously on both sides, minimizing room for error. Further discussion: https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=4.%20Listings%20should%20alternate%20between%20paths%20to%20minimize%20errors	2024-01-20 14:50:08 -05:00
nielash	0cac5d67ab	bisync: introduce terminal colors ... This introduces a few basic color codings to make the terminal output more readable (and more fun). Rclone's standard --color flag is supported. (AUTO|NEVER|ALWAYS) Only a few lines have colors right now -- more will probably be added in future versions.	2024-01-20 14:50:08 -05:00
nielash	6d6dc00abb	bisync: rollback listing on error ... Before this change, bisync had no mechanism for "retrying" a file again next time, in the event of an unexpected and possibly temporary error. After this change, bisync is now essentially able to mark a file as needing to be rechecked next time. Bisync does this by keeping one prior listing on hand at all times. In a low-confidence situation, bisync can revert a given file row back to its state at the end of the last known successful sync, ensuring that any subsequent changes will be re-noticed on the next run. This can potentially be helpful for a dynamically changing file system, where files may be changing quickly while bisync is working with them.	2024-01-20 14:50:08 -05:00
nielash	5ca61ab705	bisync: equality check before renaming (leave identical files alone) ... Improved detection of false positive change conflicts (identical files are now left alone instead of renamed) https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=1.%20Identical%20files%20should%20be%20left%20alone%2C%20even%20if%20new/newer/changed%20on%20both%20sides	2023-08-12 17:24:21 +01:00