github-mirror/rclone
2
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2024-12-23 12:13:48 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
8d3bcc025a
rclone/cmd/bisync/testdata/test_resolve/golden/test.log

305 lines
17 KiB
Plaintext
Raw Normal View History

bisync: add options to auto-resolve conflicts - fixes #7471 Before this change, when a file was new/changed on both paths (relative to the prior sync), and the versions on each side were not identical, bisync would keep both versions, renaming them with ..path1 and ..path2 suffixes, respectively. Many users have requested more control over how bisync handles such conflicts -- including an option to automatically select one version as the "winner" and rename or delete the "loser". This change introduces support for such options. --conflict-resolve CHOICE In bisync, a "conflict" is a file that is *new* or *changed* on *both sides* (relative to the prior run) AND is *not currently identical* on both sides. `--conflict-resolve` controls how bisync handles such a scenario. The currently supported options are: - `none` - (the default) - do not attempt to pick a winner, keep and rename both files according to `--conflict-loser` and `--conflict-suffix` settings. For example, with the default settings, `file.txt` on Path1 is renamed `file.txt.conflict1` and `file.txt` on Path2 is renamed `file.txt.conflict2`. Both are copied to the opposite path during the run, so both sides end up with a copy of both files. (As `none` is the default, it is not necessary to specify `--conflict-resolve none` -- you can just omit the flag.) - `newer` - the newer file (by `modtime`) is considered the winner and is copied without renaming. The older file (the "loser") is handled according to `--conflict-loser` and `--conflict-suffix` settings (either renamed or deleted.) For example, if `file.txt` on Path1 is newer than `file.txt` on Path2, the result on both sides (with other default settings) will be `file.txt` (winner from Path1) and `file.txt.conflict1` (loser from Path2). - `older` - same as `newer`, except the older file is considered the winner, and the newer file is considered the loser. - `larger` - the larger file (by `size`) is considered the winner (regardless of `modtime`, if any). - `smaller` - the smaller file (by `size`) is considered the winner (regardless of `modtime`, if any). - `path1` - the version from Path1 is unconditionally considered the winner (regardless of `modtime` and `size`, if any). This can be useful if one side is usually more trusted or up-to-date than the other. - `path2` - same as `path1`, except the path2 version is considered the winner. For all of the above options, note the following: - If either of the underlying remotes lacks support for the chosen method, it will be ignored and fall back to `none`. (For example, if `--conflict-resolve newer` is set, but one of the paths uses a remote that doesn't support `modtime`.) - If a winner can't be determined because the chosen method's attribute is missing or equal, it will be ignored and fall back to `none`. (For example, if `--conflict-resolve newer` is set, but the Path1 and Path2 modtimes are identical, even if the sizes may differ.) - If the file's content is currently identical on both sides, it is not considered a "conflict", even if new or changed on both sides since the prior sync. (For example, if you made a change on one side and then synced it to the other side by other means.) Therefore, none of the conflict resolution flags apply in this scenario. - The conflict resolution flags do not apply during a `--resync`, as there is no "prior run" to speak of (but see `--resync-mode` for similar options.) --conflict-loser CHOICE `--conflict-loser` determines what happens to the "loser" of a sync conflict (when `--conflict-resolve` determines a winner) or to both files (when there is no winner.) The currently supported options are: - `num` - (the default) - auto-number the conflicts by automatically appending the next available number to the `--conflict-suffix`, in chronological order. For example, with the default settings, the first conflict for `file.txt` will be renamed `file.txt.conflict1`. If `file.txt.conflict1` already exists, `file.txt.conflict2` will be used instead (etc., up to a maximum of 9223372036854775807 conflicts.) - `pathname` - rename the conflicts according to which side they came from, which was the default behavior prior to `v1.66`. For example, with `--conflict-suffix path`, `file.txt` from Path1 will be renamed `file.txt.path1`, and `file.txt` from Path2 will be renamed `file.txt.path2`. If two non-identical suffixes are provided (ex. `--conflict-suffix cloud,local`), the trailing digit is omitted. Importantly, note that with `pathname`, there is no auto-numbering beyond `2`, so if `file.txt.path2` somehow already exists, it will be overwritten. Using a dynamic date variable in your `--conflict-suffix` (see below) is one possible way to avoid this. Note also that conflicts-of-conflicts are possible, if the original conflict is not manually resolved -- for example, if for some reason you edited `file.txt.path1` on both sides, and those edits were different, the result would be `file.txt.path1.path1` and `file.txt.path1.path2` (in addition to `file.txt.path2`.) - `delete` - keep the winner only and delete the loser, instead of renaming it. If a winner cannot be determined (see `--conflict-resolve` for details on how this could happen), `delete` is ignored and the default `num` is used instead (i.e. both versions are kept and renamed, and neither is deleted.) `delete` is inherently the most destructive option, so use it only with care. For all of the above options, note that if a winner cannot be determined (see `--conflict-resolve` for details on how this could happen), or if `--conflict-resolve` is not in use, *both* files will be renamed. --conflict-suffix STRING[,STRING] `--conflict-suffix` controls the suffix that is appended when bisync renames a `--conflict-loser` (default: `conflict`). `--conflict-suffix` will accept either one string or two comma-separated strings to assign different suffixes to Path1 vs. Path2. This may be helpful later in identifying the source of the conflict. (For example, `--conflict-suffix dropboxconflict,laptopconflict`) With `--conflict-loser num`, a number is always appended to the suffix. With `--conflict-loser pathname`, a number is appended only when one suffix is specified (or when two identical suffixes are specified.) i.e. with `--conflict-loser pathname`, all of the following would produce exactly the same result: ``` --conflict-suffix path --conflict-suffix path,path --conflict-suffix path1,path2 ``` Suffixes may be as short as 1 character. By default, the suffix is appended after any other extensions (ex. `file.jpg.conflict1`), however, this can be changed with the `--suffix-keep-extension` flag (i.e. to instead result in `file.conflict1.jpg`). `--conflict-suffix` supports several *dynamic date variables* when enclosed in curly braces as globs. This can be helpful to track the date and/or time that each conflict was handled by bisync. For example: ``` --conflict-suffix {DateOnly}-conflict // result: myfile.txt.2006-01-02-conflict1 ``` All of the formats described [here](https://pkg.go.dev/time#pkg-constants) and [here](https://pkg.go.dev/time#example-Time.Format) are supported, but take care to ensure that your chosen format does not use any characters that are illegal on your remotes (for example, macOS does not allow colons in filenames, and slashes are also best avoided as they are often interpreted as directory separators.) To address this particular issue, an additional `{MacFriendlyTime}` (or just `{mac}`) option is supported, which results in `2006-01-02 0304PM`. Note that `--conflict-suffix` is entirely separate from rclone's main `--sufix` flag. This is intentional, as users may wish to use both flags simultaneously, if also using `--backup-dir`. Finally, note that the default in bisync prior to `v1.66` was to rename conflicts with `..path1` and `..path2` (with two periods, and `path` instead of `conflict`.) Bisync now defaults to a single dot instead of a double dot, but additional dots can be added by including them in the specified suffix string. For example, for behavior equivalent to the previous default, use: ``` [--conflict-resolve none] --conflict-loser pathname --conflict-suffix .path ```
2023-12-15 20:47:15 +08:00
(01) : test resolve
(02) : test initial bisync
(03) : bisync resync
INFO : Setting --ignore-listing-checksum as neither --checksum nor --compare checksum are set.
INFO : Bisyncing with Comparison Settings:
{
"Modtime": true,
"Size": true,
"Checksum": false,
"NoSlowHash": false,
"SlowHashSyncOnly": false,
"DownloadHash": false
}
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Copying unique Path2 files to Path1
INFO : - Path2 Resync is copying UNIQUE files to - Path1
INFO : - Path1 Resync is copying UNIQUE OR DIFFERING files to - Path2
INFO : Resync updating listings
INFO : Bisync successful
(04) : test changed on both paths and NOT identical - file1 (file1R, file1L)
(05) : touch-glob 2001-01-02 {datadir/} file1R.txt
(06) : copy-as {datadir/}file1R.txt {path2/} file1.txt
(07) : touch-glob 2001-03-04 {datadir/} file1L.txt
(08) : copy-as {datadir/}file1L.txt {path1/} file1.txt
(09) : test bisync run with --conflict-resolve=newer --conflict-loser=delete --conflict-suffix=potato
(10) : bisync conflict-resolve=newer conflict-loser=delete conflict-suffix=potato
INFO : Setting --ignore-listing-checksum as neither --checksum nor --compare checksum are set.
INFO : Bisyncing with Comparison Settings:
{
"Modtime": true,
"Size": true,
"Checksum": false,
"NoSlowHash": false,
"SlowHashSyncOnly": false,
"DownloadHash": false
}
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Building Path1 and Path2 listings
INFO : Path1 checking for diffs
INFO : - Path1 File changed: size (larger), time (newer) - file1.txt
INFO : Path1: 1 changes:  0 new,  1 modified,  0 deleted
INFO : (Modified:  1 newer,  0 older,  1 larger,  0 smaller)
INFO : Path2 checking for diffs
INFO : - Path2 File changed: size (larger), time (newer) - file1.txt
INFO : Path2: 1 changes:  0 new,  1 modified,  0 deleted
INFO : (Modified:  1 newer,  0 older,  1 larger,  0 smaller)
INFO : Applying changes
INFO : Checking potential conflicts...
ERROR : file1.txt: md5 differ
NOTICE: {path2String}: 1 differences found
NOTICE: {path2String}: 1 errors while checking
INFO : Finished checking the potential conflicts. 1 differences found
NOTICE: - WARNING New or changed in both paths - file1.txt
INFO : file1.txt: Path1 is newer. Path1: 2001-03-04 00:00:00 +0000 UTC, Path2: 2001-01-02 00:00:00 +0000 UTC, Difference: 1464h0m0s
INFO : file1.txt: The winner is: Path1
NOTICE: - Path2 Deleting Path2 copy - {path2/}file1.txt
INFO : - Path1 Queue copy to Path2 - file1.txt
INFO : - Path1 Do queued copies to - Path2
INFO : Updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
(11) : test changed on both paths and NOT identical - file1 (file1R, file1L)
(12) : touch-glob 2001-07-23 {datadir/} file1R.txt
(13) : copy-as {datadir/}file1R.txt {path2/} file1.txt
(14) : touch-glob 2001-08-26 {datadir/} file1L.txt
(15) : copy-as {datadir/}file1L.txt {path1/} file1.txt
(16) : test bisync run with --conflict-resolve=path2 --conflict-loser=num --conflict-suffix=dinosaur
(17) : bisync conflict-resolve=path2 conflict-loser=num conflict-suffix=dinosaur
INFO : Setting --ignore-listing-checksum as neither --checksum nor --compare checksum are set.
INFO : Bisyncing with Comparison Settings:
{
"Modtime": true,
"Size": true,
"Checksum": false,
"NoSlowHash": false,
"SlowHashSyncOnly": false,
"DownloadHash": false
}
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Building Path1 and Path2 listings
INFO : Path1 checking for diffs
INFO : - Path1 File changed: time (newer) - file1.txt
INFO : Path1: 1 changes:  0 new,  1 modified,  0 deleted
INFO : (Modified:  1 newer,  0 older)
INFO : Path2 checking for diffs
INFO : - Path2 File changed: time (newer) - file1.txt
INFO : Path2: 1 changes:  0 new,  1 modified,  0 deleted
INFO : (Modified:  1 newer,  0 older)
INFO : Applying changes
INFO : Checking potential conflicts...
ERROR : file1.txt: md5 differ
NOTICE: {path2String}: 1 differences found
NOTICE: {path2String}: 1 errors while checking
INFO : Finished checking the potential conflicts. 1 differences found
NOTICE: - WARNING New or changed in both paths - file1.txt
INFO : file1.txt: The winner is: Path2
NOTICE: - Path1 Renaming Path1 copy - {path1/}file1.txt.dinosaur1
NOTICE: - Path1 Queue copy to Path2 - {path2/}file1.txt.dinosaur1
NOTICE: - Path2 Not renaming Path2 copy, as it was determined the winner - {path2/}file1.txt
NOTICE: - Path2 Queue copy to Path1 - {path1/}file1.txt
INFO : - Path2 Do queued copies to - Path1
INFO : - Path1 Do queued copies to - Path2
INFO : Updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
(18) : test changed on both paths and NOT identical - file1 (file1R, file1L)
(19) : touch-glob 2002-07-23 {datadir/} file1R.txt
(20) : copy-as {datadir/}file1R.txt {path2/} file1.txt
(21) : touch-glob 2002-08-26 {datadir/} file1L.txt
(22) : copy-as {datadir/}file1L.txt {path1/} file1.txt
(23) : test bisync run with --conflict-resolve=larger --conflict-loser=num --conflict-suffix=apple
(24) : bisync conflict-resolve=larger conflict-loser=num conflict-suffix=apple
INFO : Setting --ignore-listing-checksum as neither --checksum nor --compare checksum are set.
INFO : Bisyncing with Comparison Settings:
{
"Modtime": true,
"Size": true,
"Checksum": false,
"NoSlowHash": false,
"SlowHashSyncOnly": false,
"DownloadHash": false
}
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Building Path1 and Path2 listings
INFO : Path1 checking for diffs
INFO : - Path1 File is new - file1.txt
INFO : Path1: 1 changes:  1 new,  0 modified,  0 deleted
INFO : Path2 checking for diffs
INFO : - Path2 File is new - file1.txt
INFO : Path2: 1 changes:  1 new,  0 modified,  0 deleted
INFO : Applying changes
INFO : Checking potential conflicts...
ERROR : file1.txt: md5 differ
NOTICE: {path2String}: 1 differences found
NOTICE: {path2String}: 1 errors while checking
INFO : Finished checking the potential conflicts. 1 differences found
NOTICE: - WARNING New or changed in both paths - file1.txt
INFO : file1.txt: Winner cannot be determined as sizes are equal. Path1: 33, Path2: 33, Difference: 0
INFO : file1.txt: A winner could not be determined.
NOTICE: - Path1 Renaming Path1 copy - {path1/}file1.txt.apple1
NOTICE: - Path1 Queue copy to Path2 - {path2/}file1.txt.apple1
NOTICE: - Path2 Renaming Path2 copy - {path2/}file1.txt.apple2
NOTICE: - Path2 Queue copy to Path1 - {path1/}file1.txt.apple2
INFO : - Path2 Do queued copies to - Path1
INFO : - Path1 Do queued copies to - Path2
INFO : Updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
(25) : test different suffixes
(26) : touch-glob 2003-07-23 {datadir/} file1R.txt
(27) : copy-as {datadir/}file1R.txt {path2/} file1.txt
(28) : touch-glob 2003-07-23 {datadir/} file1L.txt
(29) : copy-as {datadir/}file1L.txt {path1/} file1.txt
(30) : test bisync run with --conflict-resolve=older --conflict-loser=num --conflict-suffix=cloud,local
(31) : bisync conflict-resolve=older conflict-loser=num conflict-suffix=cloud,local
INFO : Setting --ignore-listing-checksum as neither --checksum nor --compare checksum are set.
INFO : Bisyncing with Comparison Settings:
{
"Modtime": true,
"Size": true,
"Checksum": false,
"NoSlowHash": false,
"SlowHashSyncOnly": false,
"DownloadHash": false
}
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Building Path1 and Path2 listings
INFO : Path1 checking for diffs
INFO : - Path1 File is new - file1.txt
INFO : Path1: 1 changes:  1 new,  0 modified,  0 deleted
INFO : Path2 checking for diffs
INFO : - Path2 File is new - file1.txt
INFO : Path2: 1 changes:  1 new,  0 modified,  0 deleted
INFO : Applying changes
INFO : Checking potential conflicts...
ERROR : file1.txt: md5 differ
NOTICE: {path2String}: 1 differences found
NOTICE: {path2String}: 1 errors while checking
INFO : Finished checking the potential conflicts. 1 differences found
NOTICE: - WARNING New or changed in both paths - file1.txt
INFO : file1.txt: Winner cannot be determined as times are equal. Path1: 2003-07-23 00:00:00 +0000 UTC, Path2: 2003-07-23 00:00:00 +0000 UTC, Difference: 0s
INFO : file1.txt: A winner could not be determined.
NOTICE: - Path1 Renaming Path1 copy - {path1/}file1.txt.cloud1
NOTICE: - Path1 Queue copy to Path2 - {path2/}file1.txt.cloud1
NOTICE: - Path2 Renaming Path2 copy - {path2/}file1.txt.local1
NOTICE: - Path2 Queue copy to Path1 - {path1/}file1.txt.local1
INFO : - Path2 Do queued copies to - Path1
INFO : - Path1 Do queued copies to - Path2
INFO : Updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
(32) : test legacy
(33) : touch-glob 2004-07-23 {datadir/} file1R.txt
(34) : copy-as {datadir/}file1R.txt {path2/} file1.txt
(35) : touch-glob 2004-07-23 {datadir/} file1L.txt
(36) : copy-as {datadir/}file1L.txt {path1/} file1.txt
(37) : test bisync run with --conflict-resolve=none --conflict-loser=pathname --conflict-suffix=.path
(38) : bisync conflict-resolve=none conflict-loser=pathname conflict-suffix=.path
INFO : Setting --ignore-listing-checksum as neither --checksum nor --compare checksum are set.
INFO : Bisyncing with Comparison Settings:
{
"Modtime": true,
"Size": true,
"Checksum": false,
"NoSlowHash": false,
"SlowHashSyncOnly": false,
"DownloadHash": false
}
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Building Path1 and Path2 listings
INFO : Path1 checking for diffs
INFO : - Path1 File is new - file1.txt
INFO : Path1: 1 changes:  1 new,  0 modified,  0 deleted
INFO : Path2 checking for diffs
INFO : - Path2 File is new - file1.txt
INFO : Path2: 1 changes:  1 new,  0 modified,  0 deleted
INFO : Applying changes
INFO : Checking potential conflicts...
ERROR : file1.txt: md5 differ
NOTICE: {path2String}: 1 differences found
NOTICE: {path2String}: 1 errors while checking
INFO : Finished checking the potential conflicts. 1 differences found
NOTICE: - WARNING New or changed in both paths - file1.txt
NOTICE: - Path1 Renaming Path1 copy - {path1/}file1.txt..path1
NOTICE: - Path1 Queue copy to Path2 - {path2/}file1.txt..path1
NOTICE: - Path2 Renaming Path2 copy - {path2/}file1.txt..path2
NOTICE: - Path2 Queue copy to Path1 - {path1/}file1.txt..path2
INFO : - Path2 Do queued copies to - Path1
INFO : - Path1 Do queued copies to - Path2
INFO : Updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
(39) : test deletes on both sides with default suffix
(40) : touch-glob 2005-01-02 {datadir/} file1R.txt
(41) : copy-as {datadir/}file1R.txt {path2/} file1.txt
(42) : copy-as {datadir/}file1R.txt {path1/} file2.txt
(43) : touch-glob 2006-03-04 {datadir/} file1L.txt
(44) : copy-as {datadir/}file1L.txt {path1/} file1.txt
(45) : copy-as {datadir/}file1L.txt {path2/} file2.txt
(46) : test bisync run with --conflict-resolve=newer --conflict-loser=delete
(47) : bisync conflict-resolve=newer conflict-loser=delete
INFO : Setting --ignore-listing-checksum as neither --checksum nor --compare checksum are set.
INFO : Bisyncing with Comparison Settings:
{
"Modtime": true,
"Size": true,
"Checksum": false,
"NoSlowHash": false,
"SlowHashSyncOnly": false,
"DownloadHash": false
}
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Building Path1 and Path2 listings
INFO : Path1 checking for diffs
INFO : - Path1 File changed: size (larger), time (newer) - file2.txt
INFO : - Path1 File is new - file1.txt
INFO : Path1: 2 changes:  1 new,  1 modified,  0 deleted
INFO : (Modified:  1 newer,  0 older,  1 larger,  0 smaller)
INFO : Path2 checking for diffs
INFO : - Path2 File changed: size (larger), time (newer) - file2.txt
INFO : - Path2 File is new - file1.txt
INFO : Path2: 2 changes:  1 new,  1 modified,  0 deleted
INFO : (Modified:  1 newer,  0 older,  1 larger,  0 smaller)
INFO : Applying changes
INFO : Checking potential conflicts...
ERROR : file1.txt: md5 differ
ERROR : file2.txt: md5 differ
NOTICE: {path2String}: 2 differences found
NOTICE: {path2String}: 2 errors while checking
INFO : Finished checking the potential conflicts. 2 differences found
NOTICE: - WARNING New or changed in both paths - file1.txt
INFO : file1.txt: Path1 is newer. Path1: 2006-03-04 00:00:00 +0000 UTC, Path2: 2005-01-02 00:00:00 +0000 UTC, Difference: 10224h0m0s
INFO : file1.txt: The winner is: Path1
NOTICE: - Path2 Deleting Path2 copy - {path2/}file1.txt
INFO : - Path1 Queue copy to Path2 - file1.txt
NOTICE: - WARNING New or changed in both paths - file2.txt
INFO : file2.txt: Path2 is newer. Path1: 2005-01-02 00:00:00 +0000 UTC, Path2: 2006-03-04 00:00:00 +0000 UTC, Difference: 10224h0m0s
INFO : file2.txt: The winner is: Path2
NOTICE: - Path1 Deleting Path1 copy - {path1/}file2.txt
INFO : - Path2 Queue copy to Path1 - file2.txt
INFO : - Path2 Do queued copies to - Path1
INFO : - Path1 Do queued copies to - Path2
INFO : Updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
Reference in New Issue Copy Permalink