mirror of
https://github.com/rclone/rclone.git
synced 2024-11-25 17:57:44 +08:00
s3: fix InvalidRequest copying to a locked bucket from a source with no MD5SUM
Before this change, we would upload files as single part uploads even if the source MD5SUM was not available. AWS won't let you upload a file to a locket bucket without some sort of hash protection of the upload which we don't have with no MD5SUM. So we switch to multipart upload when the source does not have an MD5SUM. This means that if --s3-disable-checksum is set or we are copying from a source with no MD5SUMs we will copy with multipart uploads. This patch changes all uploads, not just those to locked buckets because having no MD5SUM protection on uploads is undesirable. Fixes #6846
This commit is contained in:
parent
46484022b0
commit
e5a1bcb1ce
|
@ -1912,6 +1912,9 @@ size (e.g. from "rclone rcat" or uploaded with "rclone mount" or google
|
||||||
photos or google docs) they will be uploaded as multipart uploads
|
photos or google docs) they will be uploaded as multipart uploads
|
||||||
using this chunk size.
|
using this chunk size.
|
||||||
|
|
||||||
|
Files with no source MD5 will also be uploaded with multipart uploads
|
||||||
|
as will all files if --s3-disable-checksum is set.
|
||||||
|
|
||||||
Note that "--s3-upload-concurrency" chunks of this size are buffered
|
Note that "--s3-upload-concurrency" chunks of this size are buffered
|
||||||
in memory per transfer.
|
in memory per transfer.
|
||||||
|
|
||||||
|
@ -1967,7 +1970,11 @@ The minimum is 0 and the maximum is 5 GiB.`,
|
||||||
Normally rclone will calculate the MD5 checksum of the input before
|
Normally rclone will calculate the MD5 checksum of the input before
|
||||||
uploading it so it can add it to metadata on the object. This is great
|
uploading it so it can add it to metadata on the object. This is great
|
||||||
for data integrity checking but can cause long delays for large files
|
for data integrity checking but can cause long delays for large files
|
||||||
to start uploading.`,
|
to start uploading.
|
||||||
|
|
||||||
|
Note that setting this flag forces all uploads to be multipart uploads
|
||||||
|
as we can't protect the body of the transfer unless we have an MD5.
|
||||||
|
`,
|
||||||
Default: false,
|
Default: false,
|
||||||
Advanced: true,
|
Advanced: true,
|
||||||
}, {
|
}, {
|
||||||
|
@ -5500,6 +5507,12 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// If source MD5SUM not available then do multipart upload
|
||||||
|
// otherwise uploads are not hash protected and locked buckets
|
||||||
|
// will complain #6846
|
||||||
|
if !multipart && md5sumHex == "" {
|
||||||
|
multipart = true
|
||||||
|
}
|
||||||
|
|
||||||
// Set the content type it it isn't set already
|
// Set the content type it it isn't set already
|
||||||
if req.ContentType == nil {
|
if req.ContentType == nil {
|
||||||
|
|
Loading…
Reference in New Issue
Block a user