HADOOP-19576: Disable Purging Pending MPUs Before Directory Purge

[shameersss1]

Description of PR

Pending MPUs are aborted by default for S3 express store. This leads to job failure for use cases where the directory needs be purged before the final job commit, Hence disabling the pending MPUs purging for all types of buckets.

How was this patch tested?

Test with us-east-1 with S3 express store bucket. The following tests were failing with and without the change

ITestTreewalkProblems.testDistCp:317->lambda$testDistCp$3:318 
ITestTreewalkProblems.testDistCpNoIterator:340->lambda$testDistCpNoIterator$4:341 [Exit code of distcp -update -delete -direct 
ITestCustomSigner.testCustomSignerAndInitializer
ITestS3AContractAnalyticsStreamVectoredRead.testVectoredReadAfterNormalRead
ITestS3AEndpointRegion.testCentralEndpointAndNullRegionFipsWithCRUD:510 » AWSUnsupportedFeature
ITestS3AEndpointRegion.testCentralEndpointAndNullRegionWithCRUD:501->assertOpsUsingNewFs:548 » UnknownHost
ITestS3AEndpointRegion.testWithCrossRegionAccess:395 » UnknownHost getFileStat...
ITestS3AEndpointRegion.testWithOutCrossRegionAccess:374->lambda$testWithOutCrossRegionAccess$2:376 » UnknownHost
ITestConnectionTimeouts.testObjectUploadTimeouts:265 » AWSBadRequest Writing O...
ITestS3APutIfMatchAndIfNoneMatch.testIfMatchTwoMultipartUploadsRaceConditionOneClosesFirst:551 » AWSS3IO
ITestS3APutIfMatchAndIfNoneMatch.testIfNoneMatchConflictOnMultipartUpload:321->lambda$testIfNoneMatchConflictOnMultipartUpload$2:322->createFileWithFlags:176 » O
ITestS3APutIfMatchAndIfNoneMatch.testIfNoneMatchMultipartUploadWithRaceCondition:349 » AWSS3IO
ITestS3APutIfMatchAndIfNoneMatch.testIfNoneMatchTwoConcurrentMultipartUploads:372 » AWSS3

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	21m 30s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 9s		trunk passed
+1 💚	compile	0m 47s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 36s		trunk passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚	checkstyle	0m 33s		trunk passed
+1 💚	mvnsite	0m 42s		trunk passed
+1 💚	javadoc	0m 42s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 34s		trunk passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚	spotbugs	1m 11s		trunk passed
+1 💚	shadedclient	39m 43s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 31s		the patch passed
+1 💚	compile	0m 39s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 39s		the patch passed
+1 💚	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚	javac	0m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 22s		the patch passed
+1 💚	mvnsite	0m 33s		the patch passed
+1 💚	javadoc	0m 30s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 26s		the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚	spotbugs	1m 7s		the patch passed
+1 💚	shadedclient	39m 7s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	3m 29s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		156m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.50 ServerAPI=1.50 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7722/1/artifact/out/Dockerfile
GITHUB PR	#7722
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 723ee363572c 5.15.0-139-generic #149-Ubuntu SMP Fri Apr 11 22:06:13 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / ba9df1c
Default Java	Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7722/1/testReport/
Max. process+thread count	577 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7722/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[shameersss1]

@steveloughran - Could you please review the changes ?

[steveloughran]

cn you point to some docs about s3express and MPUs?

we had lots of pain related to directories not existing but still being found in list calls, and had to make changes across the code to cope with it. Are these now superfluous?

[steveloughran]

+1 pending you declaring what the test cli was, and what you tested against. s3 express, presumably