Massive parallel upload from AWS lambda #6448
Description
Describe the bug
Currently I need to upload several hundred files. I have a fork-join pool to handle preparation of byte arrays. Around 50-th file all upload became failing with cryptic error. Looks like some kind of resource is saturated
Caused by: software.amazon.awssdk.crt.CrtRuntimeException: S3Client.aws_s3_client_new: creating aws_s3_client failed (aws_last_error: AWS_ERROR_SYS_CALL_FAILURE(46), System call failure.)
--
at software.amazon.awssdk.crt.s3.S3Client.s3ClientNew(Native Method)
at software.amazon.awssdk.crt.s3.S3Client.<init>(S3Client.java:78)
at software.amazon.awssdk.services.s3.internal.crt.S3CrtAsyncHttpClient.<init>(S3CrtAsyncHttpClient.java:75)
at software.amazon.awssdk.services.s3.internal.crt.S3CrtAsyncHttpClient.<init>(S3CrtAsyncHttpClient.java:64)
at software.amazon.awssdk.services.s3.internal.crt.S3CrtAsyncHttpClient$Builder.build(S3CrtAsyncHttpClient.java:265)
at software.amazon.awssdk.services.s3.internal.crt.S3CrtAsyncHttpClient$Builder.buildWithDefaults(S3CrtAsyncHttpClient.java:271)
at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.lambda$resolveAsyncHttpClient$18(SdkDefaultClientBuilder.java:480)
at java.base/java.util.Optional.map(Optional.java:260)
at software.amazon.awssdk.utils.Either.lambda$map$0(Either.java:51)
at java.base/java.util.Optional.orElseGet(Optional.java:364)
at software.amazon.awssdk.utils.Either.map(Either.java:51)
at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.lambda$resolveAsyncHttpClient$19(SdkDefaultClientBuilder.java:480)
at java.base/java.util.Optional.map(Optional.java:260)
at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.resolveAsyncHttpClient(SdkDefaultClientBuilder.java:480)
at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.lambda$finalizeAsyncConfiguration$6(SdkDefaultClientBuilder.java:326)
at software.amazon.awssdk.utils.AttributeMap$DerivedValue.primeCache(AttributeMap.java:604)
at software.amazon.awssdk.utils.AttributeMap$DerivedValue.get(AttributeMap.java:593)
at software.amazon.awssdk.utils.AttributeMap$Builder.resolveValue(AttributeMap.java:400)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at software.amazon.awssdk.utils.AttributeMap$Builder.build(AttributeMap.java:362)
at software.amazon.awssdk.core.client.config.SdkClientConfiguration$Builder.build(SdkClientConfiguration.java:224)
at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.finalizeAsyncConfiguration(SdkDefaultClientBuilder.java:328)
at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.asyncClientConfiguration(SdkDefaultClientBuilder.java:238)
at software.amazon.awssdk.services.s3.DefaultS3AsyncClientBuilder.buildClient(DefaultS3AsyncClientBuilder.java:51)
at software.amazon.awssdk.services.s3.DefaultS3AsyncClientBuilder.buildClient(DefaultS3AsyncClientBuilder.java:27)
at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.build(SdkDefaultClientBuilder.java:173)
at software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient.initializeS3AsyncClient(DefaultS3CrtAsyncClient.java:149)
at software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient.<init>(DefaultS3CrtAsyncClient.java:83)
at software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient.<init>(DefaultS3CrtAsyncClient.java:77)
at software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient$DefaultS3CrtClientBuilder.build(DefaultS3CrtAsyncClient.java:313)
at software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient$DefaultS3CrtClientBuilder.build(DefaultS3CrtAsyncClient.java:184)
On my local machine it works perfectly. I tried to have semaphore with 10 concurrent upload but it didn't help.
val crtTransferManagersSemaphore = Semaphore(10)
try {
// S3Client.aws_s3_client_new: creating aws_s3_client failed (aws_last_error: AWS_ERROR_SYS_CALL_FAILURE(46), System call failure.)
// TlsContext.tls_ctx_new: Failed to create new aws_tls_ctx (aws_last_error: AWS_IO_TLS_CTX_ERROR(1033), Failed to create tls context)
if (!crtTransferManagersSemaphore.tryAcquire(1, TimeUnit.MINUTES))
throw RuntimeException("Failed to acquire semaphore for CRT Upload")
try {
getCrtTransferManager().use { tm ->
val result = tm.upload { uf ->
uf.putObjectRequest { po ->
po.bucket(bucket)
po.key(key)
if (!mimeContentType.isNullOrEmpty()) {
po.contentType(mimeContentType)
}
if (kmsId != null) {
po.ssekmsKeyId(kmsId)
po.serverSideEncryption(ServerSideEncryption.AWS_KMS)
}
}
uf.addTransferListener(LoggingTransferListener.create())
uf.requestBody(asyncRequestBody)
}.completionFuture().join()
result.response().eTag()
}
} finally {
crtTransferManagersSemaphore.release()
}
} catch (e: Exception) {
throw RuntimeException("Failed to upload", e)
}
Regression Issue
- Select this option if this issue appears to be a regression.
Expected Behavior
Files uploaded correctly
Current Behavior
S3Client.aws_s3_client_new: creating aws_s3_client failed (aws_last_error: AWS_ERROR_SYS_CALL_FAILURE(46), System call failure.)
TlsContext.tls_ctx_new: Failed to create new aws_tls_ctx (aws_last_error: AWS_IO_TLS_CTX_ERROR(1033), Failed to create tls context)
Reproduction Steps
Create a several hundred byte arrays on aws lambda and try to upload
Possible Solution
No response
Additional Information/Context
No response
AWS Java SDK version used
aws 2.28.18 aws-crt 0.31.3
JDK version used
public.ecr.aws/amazoncorretto/amazoncorretto:17-al2023-headless
Operating System and version
public.ecr.aws/amazoncorretto/amazoncorretto:17-al2023-headless