s3: fix InvalidRequest copying to a locked bucket from a source with no MD5SUM

Before this change, we would upload files as single part uploads even
if the source MD5SUM was not available.

AWS won't let you upload a file to a locket bucket without some sort
of hash protection of the upload which we don't have with no MD5SUM.

So we switch to multipart upload when the source does not have an
MD5SUM.

This means that if --s3-disable-checksum is set or we are copying from
a source with no MD5SUMs we will copy with multipart uploads.

This patch changes all uploads, not just those to locked buckets
because having no MD5SUM protection on uploads is undesirable.

Fixes #6846
This commit is contained in:
Nick Craig-Wood 2023-03-16 17:49:11 +00:00
parent 46484022b0
commit e5a1bcb1ce

View File

@ -1912,6 +1912,9 @@ size (e.g. from "rclone rcat" or uploaded with "rclone mount" or google
photos or google docs) they will be uploaded as multipart uploads
using this chunk size.
Files with no source MD5 will also be uploaded with multipart uploads
as will all files if --s3-disable-checksum is set.
Note that "--s3-upload-concurrency" chunks of this size are buffered
in memory per transfer.
@ -1967,7 +1970,11 @@ The minimum is 0 and the maximum is 5 GiB.`,
Normally rclone will calculate the MD5 checksum of the input before
uploading it so it can add it to metadata on the object. This is great
for data integrity checking but can cause long delays for large files
to start uploading.`,
to start uploading.
Note that setting this flag forces all uploads to be multipart uploads
as we can't protect the body of the transfer unless we have an MD5.
`,
Default: false,
Advanced: true,
}, {
@ -5500,6 +5507,12 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
}
}
}
// If source MD5SUM not available then do multipart upload
// otherwise uploads are not hash protected and locked buckets
// will complain #6846
if !multipart && md5sumHex == "" {
multipart = true
}
// Set the content type it it isn't set already
if req.ContentType == nil {