github-mirror/rclone
2
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2024-12-03 14:03:41 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
7,572 Commits 177 Branches 114 Tags 344 MiB
f45cee831f
Commit Graph

7572 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Craig-Wood
e6fde67491 s3: fix retry logic, logging and error reporting for chunk upload 
- move retries into correct place into lowest level functions
- fix logging and error reporting
 2023-08-24 12:39:27 +01:00
Nick Craig-Wood
b4e3332e02 fs: introduces aliases for OpenWriterAtFn and OpenChunkWriterFn 2023-08-24 12:39:27 +01:00
Nick Craig-Wood
0dea83a4aa pool: add page backed reader/writer for multi thread uploads 2023-08-24 12:39:27 +01:00
Nick Craig-Wood
e8f3f98aa0 lib/readers: add NoSeeker to adapt io.Reader to io.ReadSeeker 2023-08-24 12:39:27 +01:00
Nick Craig-Wood
d61328e459 serve ftp: fix race condition when using the auth proxy 
In this commit we introduced a race condition when using the auth
proxy.

94a320f23c serve ftp: update to goftp.io/server v2.0.1

This was due to the re-organisation of the upstream library which made
the driver be a singleton rather than per session.

This means that when using the auth proxy we need to keep track of
which VFS to use by based on which FTP user is connected.

This also adjusts the locking so that the methods will run
concurrently.
 2023-08-23 15:11:47 +01:00
r-ricci
9844704567
docs: remove contributor's old email 2023-08-23 12:31:48 +01:00
Nick Craig-Wood
94a320f23c serve ftp: update to goftp.io/server v2.0.1 - fixes #7237 2023-08-22 17:24:05 +01:00
Nick Craig-Wood
7fc573db27 serve sftp: fix hash calculations with --vfs-cache-mode full 
Before this change uploading files with rclone to:

    rclone serve sftp --vfs-cache-mode full

Would return the error:

    command "md5sum XXX" failed with error: unexpected non file

This patch detects that the file is still in the VFS cache and reads
the MD5SUM from there rather from the remote.

Fixes #7241
 2023-08-22 13:18:36 +01:00
Nick Craig-Wood
af95616122 Add Roberto Ricci to contributors 2023-08-22 13:18:29 +01:00
Roberto Ricci
72f9f1e9c0 vfs: make sure struct field is aligned for atomic access 2023-08-22 12:52:13 +01:00
Roberto Ricci
91b8152321 vfs: use atomic types 2023-08-22 12:52:13 +01:00
Roberto Ricci
552b6c47ff lib: use atomic types 2023-08-22 12:52:13 +01:00
Roberto Ricci
01a155fb00 fs: use atomic types 2023-08-22 12:52:13 +01:00
Roberto Ricci
50d0597d56 cmount: use atomic types 2023-08-22 12:52:13 +01:00
Roberto Ricci
123a030441 smb: use atomic types 2023-08-22 12:52:13 +01:00
Roberto Ricci
28ceb323ee sftp: use atomic types 2023-08-22 12:52:13 +01:00
Roberto Ricci
c624dd5c3a seafile: use atomic types 2023-08-22 12:52:13 +01:00
Roberto Ricci
a56c11753a local: use atomic types 2023-08-22 12:52:13 +01:00
Roberto Ricci
4341d472aa filefabric: use atomic types 2023-08-22 12:52:13 +01:00
Roberto Ricci
b6e7148daf box: use atomic types 2023-08-22 12:52:13 +01:00
Roberto Ricci
45458f2cdb union: use atomic types 2023-08-22 12:52:13 +01:00
Nick Craig-Wood
de147b6e54 sftp: fix --sftp-ssh looking for ssh agent - fixes #7235 
Before this change if pass was empty this would attempt to connect to
the ssh agent even when using --sftp-ssh.

This patch prevents that.
 2023-08-21 17:43:02 +01:00
Nick Craig-Wood
11de137660 sftp: fix spurious warning when using --sftp-ssh 
When using --sftp-ssh we were warning about user/host/port even when
they were at their defaults.

See: #7235
 2023-08-21 17:43:02 +01:00
Nick Craig-Wood
156c372cd7 sync: fix lockup with --cutoff-mode=soft and --max-duration 
Before this change, when using --cutoff-mode=soft and --max-duration
rclone deadlocked when the cutoff limit was reached.

This was because the sync objects Pipe became full and nothing was
emptying it because the cutoff was reached.

This changes the context for putting items into the pipe to be the one
that gets cancelled when the cutoff is reached.

See: https://forum.rclone.org/t/sync-command-hanging-using-cutoff-mode-soft-with-max-duration-time-flags/40866
 2023-08-18 17:33:54 +01:00
Nick Craig-Wood
c979cde002 ftp: fix 425 "TLS session of data connection not resumed" errors 
As an extra security feature some FTP servers (eg FileZilla) require
that the data connection re-use the same TLS connection as the control
connection. This is a good thing for security.

The message "TLS session of data connection not resumed" means that it
was not done.

The problem turned out to be that rclone was re-using the TLS session
cache between concurrent connections so the resumed TLS data
connection could from any of the control connections.

This patch makes each TLS connection have its own session cache which
should fix the problem.

This also reverts the ftp library to the upstream version which now
contains all of our patches.

Fixes #7234
 2023-08-18 14:44:13 +01:00
Nick Craig-Wood
03aab1a123 rmdirs: remove directories concurrently controlled by --checkers 
See: https://forum.rclone.org/t/how-to-list-empty-directories-recursively/40995
 2023-08-18 12:05:15 +01:00
Nick Craig-Wood
dc803b572c Add hideo aoyama to contributors 2023-08-18 12:05:15 +01:00
Nick Craig-Wood
4d19042a61 Add Jacob Hands to contributors 2023-08-18 12:05:15 +01:00
hideo aoyama
923989d1d7
build: add snap installation 
I ( @boukendesho ) have volunteered to maintain the snap package so
this adds it back into the installation instructions.

It will set a `snap` tag visible in `rclone version` so we know where
it came from for support queries.
 2023-08-18 11:57:25 +01:00
sitiom
cf65e36cf3 ci: change Winget Releaser job to ubuntu-latest 2023-08-17 11:36:28 +01:00
Jacob Hands
cf5457c2cd fs: Fix transferTime not being set in JSON logs 
This was unintentionally broken in 04aa696
 2023-08-17 11:19:35 +01:00
Jacob Hands
ea4aa696a5 fs: Don't stop calculating average transfer speed until the operation is complete 
Currently, the average transfer speed will stop calculating 1 minute
after the last queued transfer completes. This causes the average to
stop calculating when checking is slow and the transfer queue becomes
empty.

This change will require all checks to complete before stopping the
average speed calculation.
 2023-08-16 21:43:24 +01:00
Nick Craig-Wood
34195fd3e8 sync: fix erroneous test in TestSyncOverlapWithFilter 
In this commit:

432d5d1e20 operations: fix overlapping check on case insensitive file systems

We introduced a test that makes no sense. This happens to pass without --fast-list and fail with it.

This removes the test.
 2023-08-13 11:42:31 +01:00
Nick Craig-Wood
40b8167ab4 Add Vitor Gomes to contributors 2023-08-13 11:42:22 +01:00
Nick Craig-Wood
e365f237f5 Add nielash to contributors 2023-08-13 11:41:24 +01:00
Nick Craig-Wood
7d449572bd Add alexia to contributors 2023-08-13 11:41:24 +01:00
Vitor Gomes
181fecaec3 multithread: refactor multithread operation to use OpenChunkWriter if available #7056 
If the feature OpenChunkWriter is not available, multithread tries to create an adapter from OpenWriterAt to OpenChunkWriter.
 2023-08-12 17:55:01 +01:00
Vitor Gomes
7701d1d33d config: add "multi-thread-chunk-size" flag #7056 2023-08-12 17:55:01 +01:00
Vitor Gomes
6dd736fbdc s3: refactor MultipartUpload to use OpenChunkWriter and ChunkWriter #7056 2023-08-12 17:55:01 +01:00
Vitor Gomes
f36ca0cd25 features: add new interfaces OpenChunkWriter and ChunkWriter #7056 2023-08-12 17:55:01 +01:00
nielash
9b3b1c7067 bisync: typo corrections & other doc improvements 2023-08-12 17:24:21 +01:00
nielash
0dd0d6a13e bisync: Add support for --create-empty-src-dirs - Fixes #6109 
Sync creation and deletion of empty directories.
https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=3.%20Bisync%20should%20create/delete%20empty%20directories%20as%20sync%20does%2C%20when%20%2D%2Dcreate%2Dempty%2Dsrc%2Ddirs%20is%20passed

Also fixed an issue causing --resync to erroneously delete empty folders and duplicate files unique to Path2
https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=2.%20%2D%2Dresync%20deletes%20data%2C%20contrary%20to%20docs
 2023-08-12 17:24:21 +01:00
nielash
e5bde42303 bisync: Add experimental --resilient mode to allow recovery from self-correctable errors 
https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=2.%20Bisync%20should%20be%20more%20resilient%20to%20self%2Dcorrectable%20errors
 2023-08-12 17:24:21 +01:00
nielash
f01a50eb47 bisync: Add new --ignore-listing-checksum flag to distinguish from --ignore-checksum 
https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=6.%20%2D%2Dignore%2Dchecksum%20should%20be%20split%20into%20two%20flags%20for%20separate%20purposes
 2023-08-12 17:24:21 +01:00
nielash
5ca61ab705 bisync: equality check before renaming (leave identical files alone) 
Improved detection of false positive change conflicts (identical files are now left alone instead of renamed)
https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=1.%20Identical%20files%20should%20be%20left%20alone%2C%20even%20if%20new/newer/changed%20on%20both%20sides
 2023-08-12 17:24:21 +01:00
nielash
4ac4ce6afd bisync: apply filters correctly during deletes 
Fixed an issue causing bisync to consider more files than necessary due to overbroad filters during delete operations
https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=5.%20Bisync%20reads%20files%20in%20excluded%20directories%20during%20delete%20operations
 2023-08-12 17:24:21 +01:00
nielash
40a874a0d8 bisync: enforce --check-access during --resync 
--check-access is now enforced during --resync, preventing data loss in certain user error scenarios
https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=%2D%2Dcheck%2Daccess%20doesn%27t%20always%20fail%20when%20it%20should
 2023-08-12 17:24:21 +01:00
nielash
f4dd86238d bisync: dry runs no longer commit filter changes 
Fixed an issue causing dry runs to inadvertently commit filter changes
https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=1.%20Dry%20runs%20are%20not%20completely%20dry
 2023-08-12 17:24:21 +01:00
nielash
4f1eafb044 gitignore: add .DS_Store and remove *.log 
.DS_Store is an irrelevant file created automatically by macOS.

*.log can't be ignored because bisync uses .log files in its test suite.
 2023-08-12 17:24:21 +01:00
alexia
20c9e0cab6 fichier: fix error code parsing 
This fixes the following error I encountered:

```
2023/08/09 16:18:49 DEBUG : failed parsing fichier error: strconv.Atoi: parsing "#374": invalid syntax
2023/08/09 16:18:49 DEBUG : pacer: low level retry 1/10 (error HTTP error 403 (403 Forbidden) returned body: "{\"status\":\"KO\",\"message\":\"Flood detected: IP Locked #374\"}")
```
 2023-08-11 00:47:01 +09:00