add option to disable integrity check for multipart upload #3007
Description
Acknowledgements
- I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
- I have verified all of my SDK modules are up-to-date (you can perform a bulk update with
go get -u github.com/aws/aws-sdk-go-v2/...)
Describe the bug
I read the announcement about the integrity check behaviour change, and tried to disable the integrity check by setting request_checksum_calculation and response_checksum_validation to when_required.
However, it seems that the integrity check cannot be disabled for multipart upload by Uploader.Upload function in feature/s3/manager package. I checked the HTTP header of the SDK's requests using tcpdump command. The results were as follows:
feature/s3/manager v1.17.57 (depending on service/s3 v1.76.0) without options:
p.H..._@PUT /5a50a3a7-c751-439f-922d-d7c4f169301a/orig_key?partNumber=1&uploadId=N2U1Y2Q0ZGYtZjFkZS00YTliLWEyMzItMmQxNzNkYzhhYjkzLmNkNWYwOTgyLTFiZjItNGE1ZC04MzZlLTVmYmRkMmQyNmZiNHgxNzM5NDA4ODUwOTg5NjYxMjI5&x-id=UploadPart HTTP/1.1
Host: localhost:35737
User-Agent: aws-sdk-go-v2/1.36.1 ua/2.1 os/linux lang/go#1.23.2 md/GOOS#linux md/GOARCH#amd64 api/s3#1.76.0 ft/s3-transfer m/E,G,Z
Content-Length: 5242880
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: c3e23ad4-e461-4c1c-a58c-0e6fc17cc14e
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=******, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-checksum-crc32;x-amz-content-sha256;x-amz-date;x-amz-sdk-checksum-algorithm, Signature=15953e6b7d081c3fbdd8537dd5b4898c59fa0dfc5967e5d885f14a7ca60628c7
Content-Type: application/octet-stream
Expect: 100-continue
X-Amz-Checksum-Crc32: m0lNkw==
X-Amz-Content-Sha256: e34d74e6bb37a93786c16bdb844ca7f465a7dbeba739ca0629ded58cb74d5a8a
X-Amz-Date: 20250213T010730Z
X-Amz-Sdk-Checksum-Algorithm: CRC32
feature/s3/manager v1.17.57 (depending on service/s3 v1.76.0) with options:
X-Amz-Checksum-Crc32 and X-Amz-Sdk-Checksum-Algorithm headers still exist.
p..o....PUT /b107bc8e-3fa1-48cc-b91c-6dc8da2f5a42/orig_key?partNumber=1&uploadId=ZjQwMzA0NzMtOWM4Zi00OGM1LWI5MzktYTE0Yjg5MTRlMzE1LjNjY2M4OWFkLWNjOGYtNGZiYS1hOWE3LTJmYTMzNDAwYTgxZHgxNzM5NDA4NDQxMzEyNjk5OTQ0&x-id=UploadPart HTTP/1.1
Host: localhost:39767
User-Agent: aws-sdk-go-v2/1.36.1 ua/2.1 os/linux lang/go#1.23.2 md/GOOS#linux md/GOARCH#amd64 api/s3#1.76.0 ft/s3-transfer m/E,G,a
Content-Length: 5242880
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: ff0ca337-d00d-43ed-8955-01b2fe85f103
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=******, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-checksum-crc32;x-amz-content-sha256;x-amz-date;x-amz-sdk-checksum-algorithm, Signature=6548e5a0a38194a7a299d9106e0d3a1fdfadbba0dfeb20dab86aa7f0c8db0997
Content-Type: application/octet-stream
Expect: 100-continue
X-Amz-Checksum-Crc32: iC+CIQ==
X-Amz-Content-Sha256: 2b4c61869e5b7fa0acc8e94c84d4c7a41a2c1c1c8eecfe6b22b6efa35792a324
X-Amz-Date: 20250213T010041Z
X-Amz-Sdk-Checksum-Algorithm: CRC32
feature/s3/manager v1.17.45 (depending on service/s3 v1.72.0):
p..u....PUT /9bce1f40-a919-4149-bcfa-02bcb33f7d4d/orig_key?partNumber=1&uploadId=MDMxOTMxNjQtYjllNC00YTcyLThhYzktNDAyMzk1Y2ExNmU1LmU0YzdjOTFlLTM1MjMtNDM5Yi04ODY3LTQ0ODg1YzU4YjVmM3gxNzM5NDA5MDIyMTgyNTkzMzg2&x-id=UploadPart HTTP/1.1
Host: localhost:34109
User-Agent: aws-sdk-go-v2/1.36.0 ua/2.1 os/linux lang/go#1.23.2 md/GOOS#linux md/GOARCH#amd64 api/s3#1.72.0 ft/s3-transfer m/E,G
Content-Length: 5242880
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 9050aa85-a034-4a7b-8280-39339f545276
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=******, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=a69ec41b84526aabbd3248d4bfe7660794886024e0d84d70577f32985c5a3f27
Content-Type: application/octet-stream
Expect: 100-continue
X-Amz-Content-Sha256: 8644553ab2366153f25ef050e09e558948904724a46b0fece5000562e048fb5a
X-Amz-Date: 20250213T011022Z
Regression Issue
- Select this option if this issue appears to be a regression.
Expected Behavior
The integrity check can be disabled for multipart upload by Uploader.Upload() by setting both request_checksum_calculation and response_checksum_validation to when_required.
Current Behavior
The integrity check cannot be disabled for multipart upload by Uploader.Upload() even if both request_checksum_calculation and response_checksum_validation are set to when_required.
Reproduction Steps
I created a client as follows.
c = s3.NewFromConfig(cfg, s3util.WithBaseEndpoint(conf.EndpointURL), s3util.WithPathStyle(), func(options *s3.Options) {
options.RequestChecksumCalculation = aws.RequestChecksumCalculationWhenRequired
options.ResponseChecksumValidation = aws.ResponseChecksumValidationWhenRequired
})
uploader = manager.NewUploader(c, func(u *manager.Uploader) {
u.PartSize = partSize
}, manager.WithUploaderRequestOptions(func(options *s3.Options) {
options.RequestChecksumCalculation = aws.RequestChecksumCalculationWhenRequired
options.ResponseChecksumValidation = aws.ResponseChecksumValidationWhenRequired
}))Then called uploader.Upload().
params := &s3.PutObjectInput{
Bucket: &bucketName,
Body: body,
Key: &key,
Metadata: metadata,
}
_, err := c.uploader.Upload(ctx, params)Possible Solution
Although I'm not entirely sure, it seems that the X-Amz-Sdk-Checksum-Algorithm header is set regardless of the options for UploadPart() which is internally called by Uploader.Upload().
aws-sdk-go-v2/feature/s3/manager/upload.go
Line 736 in e40a4a1
Additional Information/Context
Uploader.Upload worked fine with small enough objects which resulted in the single-part upload (= PutObject).
feature/s3/manager v1.17.57 (depending on service/s3 v1.76.0) with options (small object):
p.......PUT /2248ca9e-6a3c-41d4-a9dc-4c5dc6af3638/orig_key?x-id=PutObject HTTP/1.1
Host: localhost:35953
User-Agent: aws-sdk-go-v2/1.36.1 ua/2.1 os/linux lang/go#1.23.2 md/GOOS#linux md/GOARCH#amd64 api/s3#1.76.0 ft/s3-transfer m/E,G,a
Content-Length: 10
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 7500b7f2-8e29-4ed5-a4ee-99062d5784e4
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=******, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=72486789dcd72fbfbb954e86c0df3cc56dd5cd7bddc5abf3708fc5cd341f30f6
Content-Type: application/octet-stream
X-Amz-Content-Sha256: d6cca940af04ec82989a2c9c517f7aef701ae6b233c3fed108cabc26660f7b6d
X-Amz-Date: 20250213T010236Z
AWS Go SDK V2 Module Versions Used
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.57
github.com/aws/aws-sdk-go-v2/service/s3 v1.76.0
Compiler and Version used
go version go1.23.2 linux/amd64
Operating System and version
Ubuntu 22.04.5 LTS