Fix createViewWithCustomMetadataLocation tests for cloudTest task #2405

tmater · 2025-08-20T08:44:16Z

This patch fixes the createViewWithCustomMetadataLocation tests for cloudTest tasks. The original test was generating temp directories internally, causing cloudTests to fail with BadRequestException instead of the expected ForbiddenException.

Changes:

Switched to Hadoop's Path (Java Path removes slashes, e.g. s3://bucket/path -> s3:/bucket/path)
Made base classes abstract to avoid running them
Implemented createViewWithCustomMetadataLocation to allow passing a custom location

Testing:

Verified locally

snazy · 2025-08-20T09:11:29Z

...in/java/org/apache/polaris/service/it/test/PolarisRestCatalogViewAwsIntegrationTestBase.java


 /** Runs PolarisRestCatalogViewIntegrationTest on AWS. */
-public class PolarisRestCatalogViewAwsIntegrationTest
+public abstract class PolarisRestCatalogViewAwsIntegrationTestBase


Since we're renaming this type anyway, let's better use the protocol name, not the vendor name.

Suggested change

public abstract class PolarisRestCatalogViewAwsIntegrationTestBase

public abstract class PolarisRestCatalogViewS3IntegrationTestBase

Shall I change these 3 (non-view test bases) as well in one go:

PolarisRestCatalogAwsIntegrationTestBase

PolarisRestCatalogAzureIntegrationTestBase

PolarisRestCatalogGcpIntegrationTestBase

Also the 6 implementations:

RestCatalogAwsIT

RestCatalogAzureIT

RestCatalogGcpIT

RestCatalogViewAwsIT

RestCatalogViewAzureIT

RestCatalogViewGcpIT

I'd support it, yea

Mean, we support MinIO and a bunch of other S3 compatible systems, not only AWS. Sure, we could argue about GCS or ADLS. However, GCS and ADLS are specific, where a vendor name or "product suite" name are not.

Makes sense, renamed the classes and related parts.

snazy · 2025-08-20T09:12:44Z

...in/java/org/apache/polaris/service/it/test/PolarisRestCatalogViewAwsIntegrationTestBase.java

@@ -20,13 +20,12 @@

 import java.util.List;
 import java.util.Optional;
-import java.util.stream.Stream;
+import org.apache.hadoop.fs.Path;


Hm, sure, this is just a test, but could we avoid Hadoop dependencies?

The idea behind Hadoop Path came because Iceberg does something "funky" with the metadataFileLocation() that resembles Hadoop Path behavior: It normalizes the scheme file:///... to file:/..., but it keeps s3://... as s3://....
For the assertion later I couldn't use java.nio.Path because that normalizes the scheme and creates an incorrect s3:/... path.

Just dug a bit more, I found a reference to Hadoop Path in Iceberg's LocationProviders:24, maybe this is how they do it as well, I could not find the exact location where it gets normalized.

I’m open to other approaches, but the only solution that came to mind was writing a custom assertion, which I’m also happy with.

Argh - the damn file URI scheme is so ambiguous (IMHO, because of multiple representations for the same thing, legacy representations, risk of interpreting those wrong).

If it's just about eliminating all host related parts, I'd say let's just use a regex to "fix" the paths?
Something like

protected static final Pattern FILE_LOCATION_PATTERN = Pattern.compile("file:/*(.*)"); protected String fixFileUri(String location) { var m = FILE_LOCATION_PATTERN.matcher(location); return m.matches() ? "file:/" + m.group(1) : location; }

WDYT? Would that work?

Just FYI In Polaris we have:

org.apache.polaris.core.storage.StorageLocation which standardizes file URIs to "file:///

but InMemoryStorageIntegration does the opposite 😅 :

polaris/polaris-core/src/main/java/org/apache/polaris/core/storage/InMemoryStorageIntegration.java

Line 74 in 532ee51

.map(str -> str.replace("file:///", "file:/"))

Hm - maybe let's ignore the file scheme specialty and just assume that it's "correct" if supplied.

Good point, I used java.nio.file.Path to remove the scheme.

...in/java/org/apache/polaris/service/it/test/PolarisRestCatalogViewGcpIntegrationTestBase.java

.../java/org/apache/polaris/service/it/test/PolarisRestCatalogViewAzureIntegrationTestBase.java

This patch fixes the createViewWithCustomMetadataLocation tests for cloudTest tasks. The original test was generating temp directories internally, causing cloudTests to fail with BadRequestException instead of the expected ForbiddenException. Changes: - Switched to Hadoop's Path (Java Path removes slashes, e.g. s3://bucket/path -> s3:/bucket/path) - Made base classes abstract to avoid running them - Implemented createViewWithCustomMetadataLocation to allow passing a custom location Testing: - Verified locally

adutra · 2025-08-28T14:45:04Z

.../src/main/java/org/apache/polaris/service/it/test/PolarisRestCatalogViewIntegrationBase.java

+  protected String getCustomMetadataLocationDir() {
+    return "";
+  }


Suggested change

protected String getCustomMetadataLocationDir() {

return "";

}

protected abstract String getCustomMetadataLocationDir();

adutra · 2025-08-28T14:45:46Z

.../src/main/java/org/apache/polaris/service/it/test/PolarisRestCatalogViewIntegrationBase.java

@@ -86,6 +86,8 @@ public abstract class PolarisRestCatalogViewIntegrationBase extends ViewCatalogT
  private static PolarisApiEndpoints endpoints;
  private static PolarisClient client;
  private static ManagementApi managementApi;
+  protected static final String POLARIS_IT_SUBDIR = "polaris_it";
+  protected static final String POLARIS_IT_CUSTOM_SUBDIR = "polaris_it_custom";


This constant seems unused.

adutra · 2025-08-28T14:51:57Z

...n/java/org/apache/polaris/service/it/test/PolarisRestCatalogViewAdlsIntegrationTestBase.java

-  protected boolean shouldSkip() {
-    return Stream.of(BASE_LOCATION, TENANT_ID).anyMatch(Strings::isNullOrEmpty);
+  protected String getCustomMetadataLocationDir() {
+    return StorageUtil.concatFilePrefixes(BASE_LOCATION, POLARIS_IT_SUBDIR, File.separator);


Shouldn't this be:

Suggested change

return StorageUtil.concatFilePrefixes(BASE_LOCATION, POLARIS_IT_SUBDIR, File.separator);

return StorageUtil.concatFilePrefixes(BASE_LOCATION, POLARIS_IT_CUSTOM_SUBDIR, File.separator);

github-project-automation bot added this to Basic Kanban Board Aug 20, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Aug 20, 2025

tmater marked this pull request as ready for review August 20, 2025 09:00

snazy reviewed Aug 20, 2025

View reviewed changes

tmater force-pushed the cloud_test_fix branch from 1c593b6 to 5a637e7 Compare August 24, 2025 18:21

tmater added 3 commits August 28, 2025 09:42

Addressing review comments, rebase

f17efa6

Self-review

c34120f

tmater force-pushed the cloud_test_fix branch from 59a0d79 to c34120f Compare August 28, 2025 09:20

adutra reviewed Aug 28, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix createViewWithCustomMetadataLocation tests for cloudTest task #2405

Fix createViewWithCustomMetadataLocation tests for cloudTest task #2405

Uh oh!

tmater commented Aug 20, 2025

Uh oh!

snazy Aug 20, 2025

Uh oh!

tmater Aug 20, 2025

Uh oh!

snazy Aug 20, 2025

Uh oh!

snazy Aug 20, 2025

Uh oh!

tmater Aug 24, 2025 •

edited

Loading

Uh oh!

snazy Aug 20, 2025

Uh oh!

tmater Aug 20, 2025

Uh oh!

snazy Aug 20, 2025

Uh oh!

adutra Aug 20, 2025 •

edited

Loading

Uh oh!

snazy Aug 20, 2025

Uh oh!

tmater Aug 24, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

adutra Aug 28, 2025

Uh oh!

adutra Aug 28, 2025

Uh oh!

adutra Aug 28, 2025

Uh oh!

Uh oh!

	public abstract class PolarisRestCatalogViewAwsIntegrationTestBase
	public abstract class PolarisRestCatalogViewS3IntegrationTestBase

	return StorageUtil.concatFilePrefixes(BASE_LOCATION, POLARIS_IT_SUBDIR, File.separator);
	return StorageUtil.concatFilePrefixes(BASE_LOCATION, POLARIS_IT_CUSTOM_SUBDIR, File.separator);

Fix createViewWithCustomMetadataLocation tests for cloudTest task #2405

Are you sure you want to change the base?

Fix createViewWithCustomMetadataLocation tests for cloudTest task #2405

Uh oh!

Conversation

tmater commented Aug 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmater Aug 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adutra Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmater Aug 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tmater Aug 24, 2025 •

edited

Loading

adutra Aug 20, 2025 •

edited

Loading

tmater Aug 24, 2025 •

edited

Loading