rclone/backend
Nick Craig-Wood 0537791d14 sftp: Fix performance regression by re-enabling concurrent writes #5197
Betweeen rclone v1.54 and v1.55 there was an approx 3x performance
regression when transferring to distant SFTP servers (in particular
rsync.net).

This turned out to be due to the library github.com/pkg/sftp rclone
uses. Concurrent writes used to be enabled in this library by default
(for v1.12.0 as used in rclone v1.54) but they are no longer enabled
(for v1.13.0 as used in rclone v1.55) for safety reasons and it is
necessary to enable them specifically.

The safety concerns are due to the uncertainty as to whether writes
come in order and whether a half completed file might have holes in
it. This isn't a problem for rclone since a) it doesn't restart
uploads and b) it has a post-transfer checksum test.

This change introduces a new flag `--sftp-disable-concurrent-writes`
to control the feature which defaults to false, meaning that
concurrent writes are enabled as in v1.54.

However this isn't quite enough to fix the problem as the sftp library
needs to be able to sniff the size of the stream from the reader
passed in, so this also adds a `Size` interface to the reader to
enable this. This involved a patch to the library.

The library was reverted to v1.12.0 for v1.55.1 - this patch installs
v1.13.0+master to fix the Size interface problem.

See: https://github.com/pkg/sftp/issues/426
2021-04-26 09:24:28 +01:00
..
alias config: make config file system pluggable 2021-03-11 17:29:26 +00:00
all new backend: uptobox 2021-04-13 17:46:07 +02:00
amazonclouddrive Fix excessive retries missing --max-duration timeout - fixes #4504 2021-03-13 09:25:44 +00:00
azureblob azureblob: add container public access level support. Fixes #5045 2021-03-15 17:18:47 +00:00
b2 b2: factor version handling into lib/version 2021-04-12 15:59:18 +01:00
box box: use upload preflight check to avoid listings in file uploads 2021-04-25 11:45:44 +01:00
cache config: make config file system pluggable 2021-03-11 17:29:26 +00:00
chunker chunker: refactor to use fspath.SplitFs instead of fspath.Parse #4996 2021-03-15 19:22:07 +00:00
compress compress: fix compressed name regexp 2021-04-22 18:38:38 +02:00
crypt crypt: support timestamped filenames from --b2-versions 2021-04-12 15:59:18 +01:00
drive drive: don't open browser when service account... 2021-04-17 19:49:53 +01:00
dropbox dropbox: fix Unable to decrypt returned paths from changeNotify - fixes #5165 2021-04-21 10:52:05 +01:00
fichier fichier: check if more than one upload link is returned #5152 2021-04-03 15:00:50 +01:00
filefabric Fix excessive retries missing --max-duration timeout - fixes #4504 2021-03-13 09:25:44 +00:00
ftp backend/ftp: fix implicit TLS after PR #4266 (#5219) 2021-04-09 01:43:50 +03:00
googlecloudstorage Fix excessive retries missing --max-duration timeout - fixes #4504 2021-03-13 09:25:44 +00:00
googlephotos Fix excessive retries missing --max-duration timeout - fixes #4504 2021-03-13 09:25:44 +00:00
hdfs hdfs: fix permissions for when directory is created 2021-03-12 09:15:14 +00:00
http config: make config file system pluggable 2021-03-11 17:29:26 +00:00
hubic swift: update github.com/ncw/swift to v2.0.0 2021-02-03 20:23:37 +00:00
jottacloud Fix excessive retries missing --max-duration timeout - fixes #4504 2021-03-13 09:25:44 +00:00
koofr fshttp: read config from ctx not passed in ConfigInfo #4685 2020-11-26 16:40:12 +00:00
local local: don't ignore preallocate disk full errors 2021-03-15 19:22:07 +00:00
mailru backends: Add context checking to remaining backends #4504 2021-03-16 16:17:22 +00:00
mega backends: Add context checking to remaining backends #4504 2021-03-16 16:17:22 +00:00
memory azureblob,memory,pcloud: fix setting of mime types 2020-11-27 14:40:05 +00:00
onedrive onedrive: Work around for random "Unable to initialize RPS" errors 2021-04-24 23:05:34 +01:00
opendrive Fix excessive retries missing --max-duration timeout - fixes #4504 2021-03-13 09:25:44 +00:00
pcloud lib/rest: fix multipart uploads stopping on context cancel 2021-03-29 19:09:47 +01:00
premiumizeme Fix excessive retries missing --max-duration timeout - fixes #4504 2021-03-13 09:25:44 +00:00
putio Fix excessive retries missing --max-duration timeout - fixes #4504 2021-03-13 09:25:44 +00:00
qingstor qingstor: fix rclone cleanup 2021-01-21 17:35:31 +00:00
s3 s3: fix --s3-profile which wasn't working - fixes #4757 2021-03-16 16:25:07 +00:00
seafile lib/rest: fix multipart uploads stopping on context cancel 2021-03-29 19:09:47 +01:00
sftp sftp: Fix performance regression by re-enabling concurrent writes #5197 2021-04-26 09:24:28 +01:00
sharefile Fix excessive retries missing --max-duration timeout - fixes #4504 2021-03-13 09:25:44 +00:00
sugarsync Fix excessive retries missing --max-duration timeout - fixes #4504 2021-03-13 09:25:44 +00:00
swift swift: implement copying large objects 2021-03-24 08:56:39 +00:00
tardigrade build: remove go1.13 build constraints 2021-02-03 17:46:08 +00:00
union union: fix initialisation broken in refactor - fixes #5139 2021-03-24 09:47:38 +00:00
uptobox new backend: uptobox 2021-04-13 17:46:07 +02:00
webdav webdav: fix sharepoint auth over http - fixes #4418 2021-04-10 11:59:56 +01:00
yandex Fix excessive retries missing --max-duration timeout - fixes #4504 2021-03-13 09:25:44 +00:00
zoho zoho: do not ask for mountpoint twice when using headless setup 2021-04-08 00:23:27 +02:00