github-mirror/rclone
2
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2024-11-22 22:22:21 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

s3: reduce the default --s3-copy-cutoff to < 5GB

Browse Source 
The maximum value for the --s3--copy-cutoff should be 5GiB as tested
with AWS S3.

However b2 have implemented this as 5GB rather than 5GiB so having the
default at 5 GiB makes the b2s3 server side copy of a large file by
default.

This patch sets the default to 4768 MiB which is slightly less than
5GB.

This should have very little effect on anything.

If in future rclone can lower this limit more if Copy can multithread.

See: https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/76
This commit is contained in:
Nick Craig-Wood 2020-09-01 18:53:29 +01:00
parent b7dd3ce608
commit 725ae91387
1 changed files with 6 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
backend/s3/s3.go
View File

@ -1064,10 +1064,12 @@ This option controls how often unused buffers will be removed from the pool.`,
// Constants // Constants
const ( const (
metaMtime = "Mtime" // the meta key to store mtime in - eg X-Amz-Meta-Mtime metaMtime = "Mtime" // the meta key to store mtime in - eg X-Amz-Meta-Mtime
metaMD5Hash = "Md5chksum" // the meta key to store md5hash in metaMD5Hash = "Md5chksum" // the meta key to store md5hash in
maxSizeForCopy = 5 * 1024 * 1024 * 1024 // The maximum size of object we can COPY // The maximum size of object we can COPY - this should be 5GiB but is < 5GB for b2 compatibility
maxUploadParts = 10000 // maximum allowed number of parts in a multi-part upload // See https://forum.rclone.org/t/copying-files-within-a-b2-bucket/16680/76
maxSizeForCopy = 4768 * 1024 * 1024
maxUploadParts = 10000 // maximum allowed number of parts in a multi-part upload
minChunkSize = fs.SizeSuffix(1024 * 1024 * 5) minChunkSize = fs.SizeSuffix(1024 * 1024 * 5)
defaultUploadCutoff = fs.SizeSuffix(200 * 1024 * 1024) defaultUploadCutoff = fs.SizeSuffix(200 * 1024 * 1024)
maxUploadCutoff = fs.SizeSuffix(5 * 1024 * 1024 * 1024) maxUploadCutoff = fs.SizeSuffix(5 * 1024 * 1024 * 1024)