rclone/cmd/bisync/testdata/test_resolve/golden/test.log
nielash 810644e873 bisync: add --resync-mode for customizing --resync - fixes #5681
Before this change, the path1 version of a file always prevailed during
--resync, and many users requested options to automatically select the winner
based on characteristics such as newer, older, larger, and smaller. This change
adds support for such options.

Note that ideally this feature would have been implemented by allowing the
existing `--resync` flag to optionally accept string values such as `--resync
newer`. However, this would have been a breaking change, as the existing flag
is a `bool` and it does not seem to be possible to have a `string` flag that
accepts both `--resync newer` and `--resync` (with no argument.) (`NoOptDefVal`
does not work for this, as it would force an `=` like `--resync=newer`.) So
instead, the best compromise to avoid a breaking change was to add a new
`--resync-mode CHOICE` flag that implies `--resync`, while maintaining the
existing behavior of `--resync` (which implies `--resync-mode path1`. i.e. both
flags are now valid, and either can be used without the other.

--resync-mode CHOICE

In the event that a file differs on both sides during a `--resync`,
`--resync-mode` controls which version will overwrite the other. The supported
options are similar to `--conflict-resolve`. For all of the following options,
the version that is kept is referred to as the "winner", and the version that
is overwritten (deleted) is referred to as the "loser". The options are named
after the "winner":

- `path1` - (the default) - the version from Path1 is unconditionally
considered the winner (regardless of `modtime` and `size`, if any). This can be
useful if one side is more trusted or up-to-date than the other, at the time of
the `--resync`.
- `path2` - same as `path1`, except the path2 version is considered the winner.
- `newer` - the newer file (by `modtime`) is considered the winner, regardless
of which side it came from. This may result in having a mix of some winners
from Path1, and some winners from Path2. (The implementation is analagous to
running `rclone copy --update` in both directions.)
- `older` - same as `newer`, except the older file is considered the winner,
and the newer file is considered the loser.
- `larger` - the larger file (by `size`) is considered the winner (regardless
of `modtime`, if any). This can be a useful option for remotes without
`modtime` support, or with the kinds of files (such as logs) that tend to grow
but not shrink, over time.
- `smaller` - the smaller file (by `size`) is considered the winner (regardless
of `modtime`, if any).

For all of the above options, note the following:
- If either of the underlying remotes lacks support for the chosen method, it
will be ignored and will fall back to the default of `path1`. (For example, if
`--resync-mode newer` is set, but one of the paths uses a remote that doesn't
support `modtime`.)
- If a winner can't be determined because the chosen method's attribute is
missing or equal, it will be ignored, and bisync will instead try to determine
whether the files differ by looking at the other `--compare` methods in effect.
(For example, if `--resync-mode newer` is set, but the Path1 and Path2 modtimes
are identical, bisync will compare the sizes.) If bisync concludes that they
differ, preference is given to whichever is the "source" at that moment. (In
practice, this gives a slight advantage to Path2, as the 2to1 copy comes before
the 1to2 copy.) If the files _do not_ differ, nothing is copied (as both sides
are already correct).
- These options apply only to files that exist on both sides (with the same
name and relative path). Files that exist *only* on one side and not the other
are *always* copied to the other, during `--resync` (this is one of the main
differences between resync and non-resync runs.).
- `--conflict-resolve`, `--conflict-loser`, and `--conflict-suffix` do not
apply during `--resync`, and unlike these flags, nothing is renamed during
`--resync`. When a file differs on both sides during `--resync`, one version
always overwrites the other (much like in `rclone copy`.) (Consider using
`--backup-dir` to retain a backup of the losing version.)
- Unlike for `--conflict-resolve`, `--resync-mode none` is not a valid option
(or rather, it will be interpreted as "no resync", unless `--resync` has also
been specified, in which case it will be ignored.)
- Winners and losers are decided at the individual file-level only (there is
not currently an option to pick an entire winning directory atomically,
although the `path1` and `path2` options typically produce a similar result.)
- To maintain backward-compatibility, the `--resync` flag implies
`--resync-mode path1` unless a different `--resync-mode` is explicitly
specified. Similarly, all `--resync-mode` options (except `none`) imply
`--resync`, so it is not necessary to use both the `--resync` and
`--resync-mode` flags simultaneously -- either one is sufficient without the
other.
2024-01-20 17:17:01 -05:00

17 KiB

(01) : test resolve
 

(02) : test initial bisync
(03) : bisync resync
INFO : Setting --ignore-listing-checksum as neither --checksum nor --compare checksum are set.
INFO : Bisyncing with Comparison Settings:
{
"Modtime": true,
"Size": true,
"Checksum": false,
"NoSlowHash": false,
"SlowHashSyncOnly": false,
"DownloadHash": false
}
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Copying Path2 files to Path1
INFO : - Path2 Resync is copying files to - Path1
INFO : - Path1 Resync is copying files to - Path2
INFO : Resync updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
 
(04) : test changed on both paths and NOT identical - file1 (file1R, file1L)
(05) : touch-glob 2001-01-02 {datadir/} file1R.txt
(06) : copy-as {datadir/}file1R.txt {path2/} file1.txt
(07) : touch-glob 2001-03-04 {datadir/} file1L.txt
(08) : copy-as {datadir/}file1L.txt {path1/} file1.txt
 
(09) : test bisync run with --conflict-resolve=newer --conflict-loser=delete --conflict-suffix=potato
 
(10) : bisync conflict-resolve=newer conflict-loser=delete conflict-suffix=potato
INFO : Setting --ignore-listing-checksum as neither --checksum nor --compare checksum are set.
INFO : Bisyncing with Comparison Settings:
{
"Modtime": true,
"Size": true,
"Checksum": false,
"NoSlowHash": false,
"SlowHashSyncOnly": false,
"DownloadHash": false
}
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Building Path1 and Path2 listings
INFO : Path1 checking for diffs
INFO : - Path1 File changed: size (larger), time (newer) - file1.txt
INFO : Path1: 1 changes: 0 new, 1 modified, 0 deleted
INFO : (Modified: 1 newer, 0 older, 1 larger, 0 smaller)
INFO : Path2 checking for diffs
INFO : - Path2 File changed: size (larger), time (newer) - file1.txt
INFO : Path2: 1 changes: 0 new, 1 modified, 0 deleted
INFO : (Modified: 1 newer, 0 older, 1 larger, 0 smaller)
INFO : Applying changes
INFO : Checking potential conflicts...
ERROR : file1.txt: md5 differ
NOTICE: {path2String}: 1 differences found
NOTICE: {path2String}: 1 errors while checking
INFO : Finished checking the potential conflicts. 1 differences found
NOTICE: - WARNING New or changed in both paths - file1.txt
INFO : file1.txt: Path1 is newer. Path1: 2001-03-04 00:00:00 +0000 UTC, Path2: 2001-01-02 00:00:00 +0000 UTC, Difference: 1464h0m0s
INFO : file1.txt: The winner is: Path1
NOTICE: - Path2 Deleting Path2 copy - {path2/}file1.txt
INFO : - Path1 Queue copy to Path2 - file1.txt
INFO : - Path1 Do queued copies to - Path2
INFO : Updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
 
(11) : test changed on both paths and NOT identical - file1 (file1R, file1L)
(12) : touch-glob 2001-07-23 {datadir/} file1R.txt
(13) : copy-as {datadir/}file1R.txt {path2/} file1.txt
(14) : touch-glob 2001-08-26 {datadir/} file1L.txt
(15) : copy-as {datadir/}file1L.txt {path1/} file1.txt
 
(16) : test bisync run with --conflict-resolve=path2 --conflict-loser=num --conflict-suffix=dinosaur
 
(17) : bisync conflict-resolve=path2 conflict-loser=num conflict-suffix=dinosaur
INFO : Setting --ignore-listing-checksum as neither --checksum nor --compare checksum are set.
INFO : Bisyncing with Comparison Settings:
{
"Modtime": true,
"Size": true,
"Checksum": false,
"NoSlowHash": false,
"SlowHashSyncOnly": false,
"DownloadHash": false
}
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Building Path1 and Path2 listings
INFO : Path1 checking for diffs
INFO : - Path1 File changed: time (newer) - file1.txt
INFO : Path1: 1 changes: 0 new, 1 modified, 0 deleted
INFO : (Modified: 1 newer, 0 older)
INFO : Path2 checking for diffs
INFO : - Path2 File changed: time (newer) - file1.txt
INFO : Path2: 1 changes: 0 new, 1 modified, 0 deleted
INFO : (Modified: 1 newer, 0 older)
INFO : Applying changes
INFO : Checking potential conflicts...
ERROR : file1.txt: md5 differ
NOTICE: {path2String}: 1 differences found
NOTICE: {path2String}: 1 errors while checking
INFO : Finished checking the potential conflicts. 1 differences found
NOTICE: - WARNING New or changed in both paths - file1.txt
INFO : file1.txt: The winner is: Path2
NOTICE: - Path1 Renaming Path1 copy - {path1/}file1.txt.dinosaur1
NOTICE: - Path1 Queue copy to Path2 - {path2/}file1.txt.dinosaur1
NOTICE: - Path2 Not renaming Path2 copy, as it was determined the winner - {path2/}file1.txt
NOTICE: - Path2 Queue copy to Path1 - {path1/}file1.txt
INFO : - Path2 Do queued copies to - Path1
INFO : - Path1 Do queued copies to - Path2
INFO : Updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
 
(18) : test changed on both paths and NOT identical - file1 (file1R, file1L)
(19) : touch-glob 2002-07-23 {datadir/} file1R.txt
(20) : copy-as {datadir/}file1R.txt {path2/} file1.txt
(21) : touch-glob 2002-08-26 {datadir/} file1L.txt
(22) : copy-as {datadir/}file1L.txt {path1/} file1.txt
 
(23) : test bisync run with --conflict-resolve=larger --conflict-loser=num --conflict-suffix=apple
 
(24) : bisync conflict-resolve=larger conflict-loser=num conflict-suffix=apple
INFO : Setting --ignore-listing-checksum as neither --checksum nor --compare checksum are set.
INFO : Bisyncing with Comparison Settings:
{
"Modtime": true,
"Size": true,
"Checksum": false,
"NoSlowHash": false,
"SlowHashSyncOnly": false,
"DownloadHash": false
}
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Building Path1 and Path2 listings
INFO : Path1 checking for diffs
INFO : - Path1 File is new - file1.txt
INFO : Path1: 1 changes: 1 new, 0 modified, 0 deleted
INFO : Path2 checking for diffs
INFO : - Path2 File is new - file1.txt
INFO : Path2: 1 changes: 1 new, 0 modified, 0 deleted
INFO : Applying changes
INFO : Checking potential conflicts...
ERROR : file1.txt: md5 differ
NOTICE: {path2String}: 1 differences found
NOTICE: {path2String}: 1 errors while checking
INFO : Finished checking the potential conflicts. 1 differences found
NOTICE: - WARNING New or changed in both paths - file1.txt
INFO : file1.txt: Winner cannot be determined as sizes are equal. Path1: 33, Path2: 33, Difference: 0
INFO : file1.txt: A winner could not be determined.
NOTICE: - Path1 Renaming Path1 copy - {path1/}file1.txt.apple1
NOTICE: - Path1 Queue copy to Path2 - {path2/}file1.txt.apple1
NOTICE: - Path2 Renaming Path2 copy - {path2/}file1.txt.apple2
NOTICE: - Path2 Queue copy to Path1 - {path1/}file1.txt.apple2
INFO : - Path2 Do queued copies to - Path1
INFO : - Path1 Do queued copies to - Path2
INFO : Updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
 
(25) : test different suffixes
(26) : touch-glob 2003-07-23 {datadir/} file1R.txt
(27) : copy-as {datadir/}file1R.txt {path2/} file1.txt
(28) : touch-glob 2003-07-23 {datadir/} file1L.txt
(29) : copy-as {datadir/}file1L.txt {path1/} file1.txt
 
(30) : test bisync run with --conflict-resolve=older --conflict-loser=num --conflict-suffix=cloud,local
 
(31) : bisync conflict-resolve=older conflict-loser=num conflict-suffix=cloud,local
INFO : Setting --ignore-listing-checksum as neither --checksum nor --compare checksum are set.
INFO : Bisyncing with Comparison Settings:
{
"Modtime": true,
"Size": true,
"Checksum": false,
"NoSlowHash": false,
"SlowHashSyncOnly": false,
"DownloadHash": false
}
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Building Path1 and Path2 listings
INFO : Path1 checking for diffs
INFO : - Path1 File is new - file1.txt
INFO : Path1: 1 changes: 1 new, 0 modified, 0 deleted
INFO : Path2 checking for diffs
INFO : - Path2 File is new - file1.txt
INFO : Path2: 1 changes: 1 new, 0 modified, 0 deleted
INFO : Applying changes
INFO : Checking potential conflicts...
ERROR : file1.txt: md5 differ
NOTICE: {path2String}: 1 differences found
NOTICE: {path2String}: 1 errors while checking
INFO : Finished checking the potential conflicts. 1 differences found
NOTICE: - WARNING New or changed in both paths - file1.txt
INFO : file1.txt: Winner cannot be determined as times are equal. Path1: 2003-07-23 00:00:00 +0000 UTC, Path2: 2003-07-23 00:00:00 +0000 UTC, Difference: 0s
INFO : file1.txt: A winner could not be determined.
NOTICE: - Path1 Renaming Path1 copy - {path1/}file1.txt.cloud1
NOTICE: - Path1 Queue copy to Path2 - {path2/}file1.txt.cloud1
NOTICE: - Path2 Renaming Path2 copy - {path2/}file1.txt.local1
NOTICE: - Path2 Queue copy to Path1 - {path1/}file1.txt.local1
INFO : - Path2 Do queued copies to - Path1
INFO : - Path1 Do queued copies to - Path2
INFO : Updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
 
(32) : test legacy
(33) : touch-glob 2004-07-23 {datadir/} file1R.txt
(34) : copy-as {datadir/}file1R.txt {path2/} file1.txt
(35) : touch-glob 2004-07-23 {datadir/} file1L.txt
(36) : copy-as {datadir/}file1L.txt {path1/} file1.txt
 
(37) : test bisync run with --conflict-resolve=none --conflict-loser=pathname --conflict-suffix=.path
 
(38) : bisync conflict-resolve=none conflict-loser=pathname conflict-suffix=.path
INFO : Setting --ignore-listing-checksum as neither --checksum nor --compare checksum are set.
INFO : Bisyncing with Comparison Settings:
{
"Modtime": true,
"Size": true,
"Checksum": false,
"NoSlowHash": false,
"SlowHashSyncOnly": false,
"DownloadHash": false
}
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Building Path1 and Path2 listings
INFO : Path1 checking for diffs
INFO : - Path1 File is new - file1.txt
INFO : Path1: 1 changes: 1 new, 0 modified, 0 deleted
INFO : Path2 checking for diffs
INFO : - Path2 File is new - file1.txt
INFO : Path2: 1 changes: 1 new, 0 modified, 0 deleted
INFO : Applying changes
INFO : Checking potential conflicts...
ERROR : file1.txt: md5 differ
NOTICE: {path2String}: 1 differences found
NOTICE: {path2String}: 1 errors while checking
INFO : Finished checking the potential conflicts. 1 differences found
NOTICE: - WARNING New or changed in both paths - file1.txt
NOTICE: - Path1 Renaming Path1 copy - {path1/}file1.txt..path1
NOTICE: - Path1 Queue copy to Path2 - {path2/}file1.txt..path1
NOTICE: - Path2 Renaming Path2 copy - {path2/}file1.txt..path2
NOTICE: - Path2 Queue copy to Path1 - {path1/}file1.txt..path2
INFO : - Path2 Do queued copies to - Path1
INFO : - Path1 Do queued copies to - Path2
INFO : Updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful
 
(39) : test deletes on both sides with default suffix
(40) : touch-glob 2005-01-02 {datadir/} file1R.txt
(41) : copy-as {datadir/}file1R.txt {path2/} file1.txt
(42) : copy-as {datadir/}file1R.txt {path1/} file2.txt
(43) : touch-glob 2006-03-04 {datadir/} file1L.txt
(44) : copy-as {datadir/}file1L.txt {path1/} file1.txt
(45) : copy-as {datadir/}file1L.txt {path2/} file2.txt
 
(46) : test bisync run with --conflict-resolve=newer --conflict-loser=delete
(47) : bisync conflict-resolve=newer conflict-loser=delete
INFO : Setting --ignore-listing-checksum as neither --checksum nor --compare checksum are set.
INFO : Bisyncing with Comparison Settings:
{
"Modtime": true,
"Size": true,
"Checksum": false,
"NoSlowHash": false,
"SlowHashSyncOnly": false,
"DownloadHash": false
}
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Building Path1 and Path2 listings
INFO : Path1 checking for diffs
INFO : - Path1 File changed: size (larger), time (newer) - file2.txt
INFO : - Path1 File is new - file1.txt
INFO : Path1: 2 changes: 1 new, 1 modified, 0 deleted
INFO : (Modified: 1 newer, 0 older, 1 larger, 0 smaller)
INFO : Path2 checking for diffs
INFO : - Path2 File changed: size (larger), time (newer) - file2.txt
INFO : - Path2 File is new - file1.txt
INFO : Path2: 2 changes: 1 new, 1 modified, 0 deleted
INFO : (Modified: 1 newer, 0 older, 1 larger, 0 smaller)
INFO : Applying changes
INFO : Checking potential conflicts...
ERROR : file2.txt: md5 differ
ERROR : file1.txt: md5 differ
NOTICE: {path2String}: 2 differences found
NOTICE: {path2String}: 2 errors while checking
INFO : Finished checking the potential conflicts. 2 differences found
NOTICE: - WARNING New or changed in both paths - file1.txt
INFO : file1.txt: Path1 is newer. Path1: 2006-03-04 00:00:00 +0000 UTC, Path2: 2005-01-02 00:00:00 +0000 UTC, Difference: 10224h0m0s
INFO : file1.txt: The winner is: Path1
NOTICE: - Path2 Deleting Path2 copy - {path2/}file1.txt
INFO : - Path1 Queue copy to Path2 - file1.txt
NOTICE: - WARNING New or changed in both paths - file2.txt
INFO : file2.txt: Path2 is newer. Path1: 2005-01-02 00:00:00 +0000 UTC, Path2: 2006-03-04 00:00:00 +0000 UTC, Difference: 10224h0m0s
INFO : file2.txt: The winner is: Path2
NOTICE: - Path1 Deleting Path1 copy - {path1/}file2.txt
INFO : - Path2 Queue copy to Path1 - file2.txt
INFO : - Path2 Do queued copies to - Path1
INFO : - Path1 Do queued copies to - Path2
INFO : Updating listings
INFO : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
INFO : Bisync successful