Add secret scopes support in assets bundling #2744
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| bundle: | ||
| name: deploy-secret-scope-azure-backend | ||
|
|
||
| resources: | ||
| secret_scopes: | ||
| secret_scope_azure: | ||
| name: test-secrets-azure-backend | ||
| backend_type: "AZURE_KEYVAULT" | ||
| keyvault_metadata: | ||
| resource_id: my_azure_keyvault_id | ||
| dns_name: my_azure_keyvault_dns_name |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/deploy-secret-scope-azure-backend/default/files... | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
|
|
||
| >>> jq -s .[] | select(.path=="/api/2.0/secrets/scopes/create") | .body out.requests.txt | ||
| { | ||
| "backend_azure_keyvault": { | ||
| "dns_name": "my_azure_keyvault_dns_name", | ||
| "resource_id": "my_azure_keyvault_id" | ||
| }, | ||
| "scope": "test-secrets-azure-backend", | ||
| "scope_backend_type": "AZURE_KEYVAULT" | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| trace $CLI bundle deploy | ||
| trace jq -s '.[] | select(.path=="/api/2.0/secrets/scopes/create") | .body' out.requests.txt | ||
| rm out.requests.txt | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| Local = true | ||
| Cloud = false | ||
|
|
||
| RecordRequests = true |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| bundle: | ||
| name: deploy-secret-scope-test-$UNIQUE_NAME | ||
anton-107 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| resources: | ||
| secret_scopes: | ||
| secret_scope1: | ||
| name: $SECRET_SCOPE_NAME | ||
| backend_type: "DATABRICKS" | ||
| permissions: | ||
| - user_name: admins | ||
| level: WRITE | ||
| - user_name: users | ||
| level: READ | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/deploy-secret-scope-test-[UNIQUE_NAME]/default/files... | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
|
|
||
| >>> [CLI] bundle summary --output json | ||
| { | ||
| "backend_type": "DATABRICKS", | ||
| "modified_status": "created", | ||
| "name": "my-secrets-[UUID]", | ||
| "permissions": [ | ||
| { | ||
| "level": "WRITE", | ||
| "user_name": "admins" | ||
| }, | ||
| { | ||
| "level": "READ", | ||
| "user_name": "users" | ||
| } | ||
| ] | ||
| } | ||
|
|
||
| >>> [CLI] secrets list-scopes -o json | ||
| { | ||
| "backend_type": "DATABRICKS", | ||
| "name": "my-secrets-[UUID]" | ||
| } | ||
|
|
||
| >>> [CLI] secrets list-acls my-secrets-[UUID] | ||
| {"permission":"MANAGE","principal":"[USERNAME]"} | ||
| {"permission":"READ","principal":"users"} | ||
| {"permission":"WRITE","principal":"admins"} | ||
|
|
||
| >>> [CLI] secrets put-secret my-secrets-[UUID] my-key --string-value my-secret-value | ||
|
|
||
| >>> [CLI] secrets get-secret my-secrets-[UUID] my-key | ||
| { | ||
| "key":"my-key", | ||
| "value":"bXktc2VjcmV0LXZhbHVl" | ||
| } | ||
|
|
||
| >>> [CLI] bundle destroy --auto-approve | ||
| The following resources will be deleted: | ||
| delete secret_acl secret_acl_secret_scope1_0 | ||
| delete secret_acl secret_acl_secret_scope1_1 | ||
| delete secret_scope secret_scope1 | ||
|
|
||
| All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/deploy-secret-scope-test-[UNIQUE_NAME]/default | ||
|
|
||
| Deleting files... | ||
| Destroy complete! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| bundle: | ||
| name: deploy-secret-scope-with-permissions | ||
|
|
||
| resources: | ||
| secret_scopes: | ||
| secret_scope_azure: | ||
| name: test-secrets-permissions | ||
|
|
||
| permissions: | ||
| - user_name: $CURRENT_USER_NAME | ||
| level: CAN_MANAGE | ||
| - group_name: users | ||
| level: CAN_VIEW | ||
| - group_name: admins | ||
| level: CAN_MANAGE |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/deploy-secret-scope-with-permissions/default/files... | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
|
|
||
| >>> jq -s -c .[] | select(.path=="/api/2.0/secrets/acls/put") | .body out.requests.txt | ||
| {"permission":"MANAGE","principal":"admins","scope":"test-secrets-permissions"} | ||
| {"permission":"READ","principal":"users","scope":"test-secrets-permissions"} | ||
|
There was a problem hiding this comment. Do you know why the current user is missing from the output here? There was a problem hiding this comment. that's because dabs silently filters out all permissions for the current user name in cli/bundle/config/mutator/resourcemutator/filter_current_user.go Lines 13 to 15 in 47d329c There was a problem hiding this comment. Can you confirm this actually leads to the desired behavior here? If I recall correctly, the integration test user is also a member of the "admins" group, so secret r/w will still be possible in the integration test, but may not be if you use a non-admin user to provision it. There was a problem hiding this comment. Secret scope api always give the caller the MANAGE permission if initial_manage_principal is omitted (and in dabs we always omit it). All the other permissions that we specify here give additional groups permission and they do not affect the permission that the caller has. This is also shown by the There was a problem hiding this comment. Thanks -- I mistakenly thought this was the output of the above, and thought the current user had their ACL stripped entirely, but this is the set of PUT requests from the client into the server. |
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| envsubst < databricks.yml.tmpl > databricks.yml | ||
| trace $CLI bundle deploy #--log-level TRACE | ||
| trace jq -s -c '.[] | select(.path=="/api/2.0/secrets/acls/put") | .body' out.requests.txt | sort | ||
| rm out.requests.txt |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| Local = true | ||
| Cloud = false | ||
| RecordRequests = true | ||
|
|
||
| [[Server]] | ||
| Pattern = "PUT /api/2.0/permissions/directories/{objectId}" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| SECRET_SCOPE_NAME="my-secrets-$(uuid)" | ||
| if [ -z "$CLOUD_ENV" ]; then | ||
| SECRET_SCOPE_NAME="my-secrets-6260d50f-e8ff-4905-8f28-812345678903" # use hard-coded uuid when running locally | ||
| fi | ||
| export SECRET_SCOPE_NAME | ||
|
|
||
| envsubst < databricks.yml.tmpl > databricks.yml | ||
|
|
||
| cleanup() { | ||
| trace $CLI bundle destroy --auto-approve | ||
| } | ||
| trap cleanup EXIT | ||
|
|
||
| trace $CLI bundle deploy | ||
| trace $CLI bundle summary --output json | jq '.resources.secret_scopes.secret_scope1' | ||
| trace $CLI secrets list-scopes -o json | jq --arg value ${SECRET_SCOPE_NAME} '.[] | select(.name == $value)' | ||
|
|
||
| trace $CLI secrets list-acls ${SECRET_SCOPE_NAME} | jq -c '.[]' | sort | ||
|
|
||
| trace $CLI secrets put-secret ${SECRET_SCOPE_NAME} my-key --string-value "my-secret-value" | ||
| trace $CLI secrets get-secret ${SECRET_SCOPE_NAME} my-key |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| Cloud = true | ||
| Local = true | ||
|
|
||
| Ignore = [ | ||
| "databricks.yml", | ||
| ] | ||
|
|
||
| [[Server]] | ||
| Pattern = "POST /api/2.0/secrets/scopes/create" | ||
|
There was a problem hiding this comment. [optional] We have CRUD implementations in test server for various resources and it's not a lot of work but makes it usable for other tests. We definitely going to need it when we add tests for terraformless implementation. https://github.com/databricks/cli/blob/main/acceptance/internal/handlers.go#L220 There was a problem hiding this comment. i suggest to do this in a follow-up PR |
||
|
|
||
| [[Server]] | ||
| Pattern = "GET /api/2.0/secrets/scopes/list" | ||
| Response.Body = ''' | ||
| { | ||
| "scopes": [ | ||
| { | ||
| "backend_type": "DATABRICKS", | ||
| "name": "my-secrets-6260d50f-e8ff-4905-8f28-812345678903" | ||
| }, | ||
| { | ||
| "backend_type": "AZURE_KEYVAULT", | ||
| "name": "test-secrets-azure-backend" | ||
| }, | ||
| { | ||
| "backend_type": "DATABRICKS", | ||
| "name": "test-secrets-permissions" | ||
| } | ||
| ] | ||
| } | ||
| ''' | ||
|
|
||
| [[Server]] | ||
| Pattern = "POST /api/2.0/secrets/scopes/delete" | ||
|
|
||
| [[Server]] | ||
| Pattern = "POST /api/2.0/secrets/put" | ||
|
|
||
| [[Server]] | ||
| Pattern = "POST /api/2.0/secrets/acls/put" | ||
|
|
||
| [[Server]] | ||
| Pattern = "GET /api/2.0/secrets/get" | ||
| Response.Body = ''' | ||
| { | ||
| "key":"my-key", | ||
| "value":"bXktc2VjcmV0LXZhbHVl" | ||
| } | ||
| ''' | ||
|
|
||
| [[Server]] | ||
| Pattern = "GET /api/2.0/secrets/acls/list" | ||
| Response.Body = ''' | ||
| { | ||
| "items": [ | ||
| { | ||
| "permission": "READ", | ||
| "principal": "users" | ||
| }, | ||
| { | ||
| "permission": "WRITE", | ||
| "principal": "admins" | ||
| }, | ||
| { | ||
| "permission": "MANAGE", | ||
| "principal": "[USERNAME]" | ||
| } | ||
| ] | ||
| } | ||
| ''' | ||
|
|
||
| [[Server]] | ||
| Pattern = "GET /api/2.0/secrets/acls/get" | ||
|
|
||
| [[Server]] | ||
| Pattern = "POST /api/2.0/secrets/acls/delete" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| bundle: | ||
| name: bind-secret-scope-test-$UNIQUE_NAME | ||
|
|
||
| resources: | ||
| secret_scopes: | ||
| secret_scope1: | ||
| name: $SECRET_SCOPE_NAME |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
|
|
||
| >>> [CLI] secrets create-scope test-secret-scope-[UUID] | ||
|
|
||
| >>> [CLI] bundle deployment bind secret_scope1 test-secret-scope-[UUID] --auto-approve | ||
| Updating deployment state... | ||
| Successfully bound secret_scope with an id 'test-secret-scope-[UUID]'. Run 'bundle deploy' to deploy changes to your workspace | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/bind-secret-scope-test-[UNIQUE_NAME]/default/files... | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
|
|
||
| >>> [CLI] secrets list-scopes -o json | ||
| { | ||
| "backend_type": "DATABRICKS", | ||
| "name": "test-secret-scope-[UUID]" | ||
| } | ||
|
|
||
| >>> [CLI] bundle deployment unbind secret_scope1 | ||
| Updating deployment state... | ||
|
|
||
| >>> [CLI] bundle destroy --auto-approve | ||
| All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/bind-secret-scope-test-[UNIQUE_NAME]/default | ||
|
|
||
| Deleting files... | ||
| Destroy complete! | ||
|
|
||
| >>> [CLI] secrets list-scopes -o json | ||
| { | ||
| "backend_type": "DATABRICKS", | ||
| "name": "test-secret-scope-[UUID]" | ||
| } | ||
|
|
||
| >>> [CLI] secrets delete-scope test-secret-scope-[UUID] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| SECRET_SCOPE_NAME="test-secret-scope-$(uuid)" | ||
| if [ -z "$CLOUD_ENV" ]; then | ||
| SECRET_SCOPE_NAME="test-secret-scope-6260d50f-e8ff-4905-8f28-812345678903" # use hard-coded uuid when running locally | ||
| fi | ||
| export SECRET_SCOPE_NAME | ||
| envsubst < databricks.yml.tmpl > databricks.yml | ||
|
|
||
| # Create a pre-defined volume: | ||
| trace $CLI secrets create-scope "${SECRET_SCOPE_NAME}" | ||
|
|
||
| cleanup() { | ||
| trace $CLI secrets delete-scope "${SECRET_SCOPE_NAME}" | ||
| } | ||
| trap cleanup EXIT | ||
|
|
||
| trace $CLI bundle deployment bind secret_scope1 "${SECRET_SCOPE_NAME}" --auto-approve | ||
|
|
||
| trace $CLI bundle deploy | ||
|
|
||
| trace $CLI secrets list-scopes -o json | jq --arg value ${SECRET_SCOPE_NAME} '.[] | select(.name == $value)' | ||
|
|
||
| trace $CLI bundle deployment unbind secret_scope1 | ||
|
|
||
| trace $CLI bundle destroy --auto-approve | ||
|
|
||
| trace $CLI secrets list-scopes -o json | jq --arg value ${SECRET_SCOPE_NAME} '.[] | select(.name == $value)' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| Local = true | ||
| Cloud = true | ||
| RequiresUnityCatalog = true | ||
|
|
||
| Ignore = [ | ||
| "databricks.yml", | ||
| ] | ||
|
|
||
| [[Server]] | ||
| Pattern = "POST /api/2.0/secrets/scopes/create" | ||
|
|
||
| [[Server]] | ||
| Pattern = "GET /api/2.0/secrets/scopes/list" | ||
| Response.Body = ''' | ||
| { | ||
| "scopes": [ | ||
| { | ||
| "backend_type": "DATABRICKS", | ||
| "name": "test-secret-scope-6260d50f-e8ff-4905-8f28-812345678903" | ||
| } | ||
| ] | ||
| } | ||
| ''' | ||
|
|
||
| [[Server]] | ||
| Pattern = "POST /api/2.0/secrets/scopes/delete" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -152,6 +152,11 @@ func mockBundle(mode config.Mode) *bundle.Bundle { | |
| }, | ||
| }, | ||
| }, | ||
| SecretScopes: map[string]*resources.SecretScope{ | ||
|
There was a problem hiding this comment. This test is a good candidate to be converted to acceptance tests (before or after this PR). There was a problem hiding this comment. i suggest to do this in a follow-up PR |
||
| "secretScope1": { | ||
| Name: "secretScope1", | ||
| }, | ||
| }, | ||
| }, | ||
| }, | ||
| SyncRoot: vfs.MustNew("/Users/[email protected]"), | ||
|
|
@@ -318,8 +323,8 @@ func TestAllNonUcResourcesAreRenamed(t *testing.T) { | |
| nameField := resource.Elem().FieldByName("Name") | ||
| resourceType := resources.Type().Field(i).Name | ||
|
|
||
| // Skip resources that are not renamed | ||
| if resourceType == "Apps" || resourceType == "SecretScopes" { | ||
| continue | ||
| } | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Although it's currently not forbidden to have nested tests, I'm thinking for making them illegal because I cannot figure out how to select one without running the other. So I'd suggest we avoid them unless you have an idea how to run outer without nested?
Example
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i suggest to do this in a follow-up PR
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, I plan to update the update all nested tests not be nested and make test runner complain about it, so if you can take care of the ones you added, that would be appreciated.