Disable custom namespace locations #2422

eric-maynard · 2025-08-21T19:59:57Z

When we create a namespace or alter its location, we must confirm that this location is within the parent location. This PR introduces introduces a check similar to the one we have for tables, where custom locations are prohibited by default. This functionality is gated behind a new behavior change flag ALLOW_NAMESPACE_CUSTOM_LOCATION. In addition to allowing us to revert to the old behavior, this flag allows some tests relying on arbitrarily-located namespaces to pass (such as those from upstream Iceberg).

Fixes: #2417

…nto namespace-location-check

…ion-check

dimas-b

LGTM overall 👍

dimas-b · 2025-08-21T23:01:19Z

polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java

@@ -100,6 +100,16 @@ public static void enforceFeatureEnabledOrThrow(
          .defaultValue(false)
          .buildFeatureConfiguration();

+  public static final FeatureConfiguration<Boolean> ALLOW_NAMESPACE_LOCATION_ESCAPE =
+      PolarisConfiguration.<Boolean>builder()
+          .key("ALLOW_NAMESPACE_LOCATION_ESCAPE")


Feature flags constitute our SemVer "API" surface, so I believe we need to formally send a discussion email to the dev ML about this flag.

Proactively +1 from my side :)

Is that so? That seems to create a weird incentive not to feature-flag anything.

In this particular case there's a behavior change due to the default setting of the flag so in that sense it's relevant to semver, but it's also a bugfix.

I'm +1 on this change already. So having extra discussion on the dev ML does not matter for me personally.

@flyrain WDYT?

+1 for the change as well. If this is more a bug fix, I don't think we even need this flag, the only reason we need the config is that existing behavior can be broken due to the change.

We could even mark it deprecated in the beginning(this PR).

@flyrain : but what's you POV about requiring dev discussion for feature flags in general?

I'm supportive on having a discussion on dev mailing list. It's great to let more people know this behavior change/bug fix.

In this case I think we want to have a flag for parity with tables -- the "bug" here IMO is that the default behavior is too lax, not that the lax behavior is accessible at all.

I'll create a quick thread to discuss the change in more detail, but in general my position is that it's not healthy for a project to require ML discussion for every new config. This creates an incentive to not protect changes with configs, and in the ideal world where all changes are config-protected this essentially necessitates an ML thread for every PR.

...ests/src/main/java/org/apache/polaris/service/it/test/PolarisApplicationIntegrationTest.java

runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java

eric-maynard · 2025-08-22T18:04:24Z

Pulling this back to draft to investigate -- we may need to just block all custom namespace locations like we do for tables in order to avoid scenarios where two namespaces overlap despite being within their parent entity

eric-maynard · 2025-08-23T06:33:06Z

I've updated the PR to make this check more like the table one, which totally disallows custom locations by default.

dennishuo · 2025-08-23T08:01:23Z

runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java

+            "Cannot create namespace without a parent storage configuration");
+      }
+      boolean allowed =
+          parentEntity.getStorageConfigurationInfo().getAllowedLocations().stream()


AllowedLocations in the StorageConfig are an independent concept vs either of:

Whether to allow "custom" locations outside of the default path hierarchy that uses namespace names

Whether to allow "overlapping" locations

The most important one is validating that anything that can vend credentials resides under the allowedLocations.

Overlap and custom hierarchies in general are more common to disable validation for.

This PR isn't related to credential vending, but the location that a namespace is created at.

Right, I was explaining why your PR description is misleading:

in order for our overlap checks and assumptions around credential vending to be valid

Is this PR fixing assumptions around credential vending, or just adding an option for whether to enforce "standard" locations for namespaces? In general we shouldn't need namespaces to be standard to still provide the intended guarantees about enforcement of ALLOWED_LOCATIONS.

The more important bug to fix is enforcement of ALLOWED_LOCATIONS.

That clause is part of a sentence:

When we create a namespace or alter its location, we must confirm that this location is within the parent location ... in order for our overlap checks and assumptions around credential vending to be valid.

and is used in that sentence to articulate why we care that namespaces reside within their parent's location. Indeed, if you remove the clause from that context it could be misleading.

If there's another (more important) bugfix, let's submit that in parallel with this existing bugfix.

My clause was also part of a paragraph:

AllowedLocations in the StorageConfig are an independent concept vs either of:

Whether to allow "custom" locations outside of the default path hierarchy that uses namespace names

Whether to allow "overlapping" locations

The most important one is validating that anything that can vend credentials resides under the allowedLocations.

Overlap and custom hierarchies in general are more common to disable validation for.

flyrain

Thanks @eric-maynard for the fix. Left some comments.

runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java

flyrain · 2025-08-26T05:30:12Z

runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java

+        FeatureConfiguration.ALLOW_NAMESPACE_CUSTOM_LOCATION, catalogEntity)) {
+      LOGGER.debug("Validating that namespace {} has a location inside its parent", namespace);
+      validateNamespaceLocation(entity, resolvedParent);
+    }


Minor: It'd be nice to wrap this into a method, so that we don't duplicate them in two different places.

if (!realmConfig.getConfig(FeatureConfiguration.ALLOW_NAMESPACE_LOCATION_OVERLAP)) { LOGGER.debug("Validating no overlap for {} with sibling tables or namespaces", namespace); validateNoLocationOverlap(entity, resolvedParent.getRawFullPath()); } else { LOGGER.debug("Skipping location overlap validation for namespace '{}'", namespace); } if (!realmConfig.getConfig( FeatureConfiguration.ALLOW_NAMESPACE_CUSTOM_LOCATION, catalogEntity)) { LOGGER.debug("Validating that namespace {} has a location inside its parent", namespace); validateNamespaceLocation(entity, resolvedParent); }

runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java

…ion-check

flyrain

@eric-maynard, thanks for moving it to BehaviorChangeConfiguration. By doing so, it discourages any users to use it. It is still useful for test purpose. We could remove it, once we figure out related tests.

Hi @dennishuo, would you take another look?

dimas-b · 2025-09-04T23:45:02Z

@eric-maynard : These tests fail on main after this PR... not sure how they passed on the PR 😅 Could you take a look?

 IcebergAllowedLocationTest > testCreateTableInsideOfCatalogAllowedLocations(Path) FAILED
    java.lang.IllegalArgumentException at IcebergAllowedLocationTest.java:352

IcebergAllowedLocationTest > testCreateTableOutSideOfCatalogAllowedLocations(Path) FAILED
    java.lang.IllegalArgumentException at IcebergAllowedLocationTest.java:352

eric-maynard · 2025-09-05T00:21:42Z

Hey @dimas-b, you're totally right, my bad. These two PRs had a conflict. I put up a fix in #2511.

I've talked to @flyrain about this in the past, but it would be pretty cool if we could have a merge queue and force PRs to pass CI before they actually merge.

dimas-b · 2025-09-05T00:24:06Z

+1 to merge queue... at least we should discuss it :)

eric-maynard added 8 commits August 21, 2025 11:35

writing test

c8aa4ba

wip

b3985fd

fixed

8f058a0

cleanups

68e83ff

Merge branch 'c2209cf0f-fix' of github.com-oss:eric-maynard/polaris i…

81e7fc6

…nto namespace-location-check

test is stable

a02197e

parameterize test

7fc8180

stable

27dc824

github-project-automation bot added this to Basic Kanban Board Aug 21, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Aug 21, 2025

eric-maynard added 2 commits August 21, 2025 13:00

Merge branch 'main' of github.com:apache/polaris into namespace-locat…

32ba495

…ion-check

fix cli test

386c608

dimas-b reviewed Aug 21, 2025

View reviewed changes

eric-maynard marked this pull request as draft August 22, 2025 18:03

eric-maynard added 2 commits August 22, 2025 11:05

Yank println

b62ab27

reworked check

b92c7b7

eric-maynard changed the title ~~Do not allow namespaces outside of allowed locations~~ Disable custom namespace locations Aug 23, 2025

eric-maynard marked this pull request as ready for review August 23, 2025 06:33

dennishuo reviewed Aug 23, 2025

View reviewed changes

eric-maynard requested review from dimas-b and flyrain August 25, 2025 22:49

eric-maynard added 2 commits August 25, 2025 16:20

better exceptions

6bc11ab

fixes for various tests

2a708f4

flyrain reviewed Aug 26, 2025

View reviewed changes

eric-maynard added 4 commits August 26, 2025 09:28

address comments

435d24a

autolint

24ea0c3

lint

23b331d

fix more tests

c0a3d2f

eric-maynard added 3 commits August 26, 2025 11:25

autolint

98ce3f9

clarify comment

43b0b95

autolint

414fc6b

flyrain previously approved these changes Aug 26, 2025

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Aug 26, 2025

eric-maynard added 4 commits August 29, 2025 11:58

change wording

a43c7aa

Merge branch 'main' of github.com:apache/polaris into namespace-locat…

9595ab1

…ion-check

pull main

6548829

fix for catalog config

d623f6b

eric-maynard dismissed flyrain’s stale review via d623f6b August 29, 2025 19:10

flyrain approved these changes Aug 29, 2025

View reviewed changes

eric-maynard merged commit 453e9fb into apache:main Sep 4, 2025
12 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Sep 4, 2025

eric-maynard mentioned this pull request Sep 5, 2025

Fix failing tests in IcebergAllowedLocationTest #2511

Merged

Disable custom namespace locations #2422

Disable custom namespace locations #2422

Uh oh!

Conversation

eric-maynard commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eric-maynard Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

eric-maynard commented Aug 22, 2025

Uh oh!

eric-maynard commented Aug 23, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flyrain left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

flyrain left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dimas-b commented Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eric-maynard commented Sep 5, 2025

Uh oh!

dimas-b commented Sep 5, 2025

Uh oh!

Uh oh!

eric-maynard commented Aug 21, 2025 •

edited

Loading

eric-maynard Aug 25, 2025 •

edited

Loading

dimas-b commented Sep 4, 2025 •

edited

Loading