From 3d51b0bb9a2d0ad828195de5e81595d01b5537a9 Mon Sep 17 00:00:00 2001 From: Patryk Kozak Date: Mon, 18 Jul 2022 14:50:25 +0200 Subject: [PATCH 1/2] Enhance the PIPELINES.md readme --- docs/PIPELINES.md | 85 +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 78 insertions(+), 7 deletions(-) diff --git a/docs/PIPELINES.md b/docs/PIPELINES.md index 5d342176..eb917017 100644 --- a/docs/PIPELINES.md +++ b/docs/PIPELINES.md @@ -1,6 +1,26 @@ Pipelines ========= +- [Pipelines](#pipelines) + * [Propagation](#propagation) + * [Configuration](#configuration) + + [Schema](#schema) + - [Pipeline YAML Field Guide](#pipeline-yaml-field-guide) + - [Secrets](#secrets) + - [Git Integration](#git-integration) + - [Locker Integration](#locker-integration) + - [Auto-Update Genesis Assets](#auto-update-genesis-assets) + - [Notifications](#notifications) + * [Notification: Slack](#notification--slack) + * [Notification: Email](#notification--email) + - [BOSH Associations](#bosh-associations) + - [Task Configuration](#task-configuration) + - [Groups](#groups) + - [Pipeline Override Configuration](#pipeline-override-configuration) + + [Useful commands](#useful-commands) + - [Retrieve **bosh** configuration](#retrieve---bosh---configuration) + - [Retrieve **vault** configuration](#retrieve---vault---configuration) + Concourse is an integral part of Genesis v2. But rather than force operators to define the full structure of a Concourse deployment pipeline (which can get rather complex), we want to provide them a domain-specific language for @@ -306,7 +326,7 @@ pipeline: #### Secrets - **pipeline.vault.url** - The URL of your Vault installation, i.e. - `https://vault.example.com`. This is **required**. + `https://vault.example.com` or `https://10.4.1.7`. This is **required**. - **pipeline.vault.role** - The AppRole GUID of a given Vault AppRole, used to generate temporary tokens for Vault-accessing during deploys. This field is @@ -544,18 +564,69 @@ being deployed, or vice versa. If you need to edit and make changes to your deployment pipeline, simply add the changes you need to the bottom of your ci.yml located in your -*-deployments repo. For example, if we have a pipeline we need to edit the +*-deployments repo to be merged by spruce. +For example, if we have a pipeline we need to edit the 'check_every:' parameter of a resource named git we can add the block below -under our pipeline layouts: +under our pipeline: **ci.yml** ``` -layouts: - test-sandbox-ops: |+ - auto *test-sandbox *test-staging - test-sandbox -> test-staging + (...) + layouts: + test-sandbox-ops: |+ + auto *test-sandbox *test-staging + test-sandbox -> test-staging + + groups: + default: + - test-sandbox + - test-staging resources: - name: git check_every: 15m ``` + +### Useful commands + +#### Retrieve **bosh** configuration +In order to fill out the `pipeline.boshes.*` section first +we need to know all of the informations. +- get bosh url +- get bosh username and password +- get bosh cert\ +All of those params are available under +```bash +cd my-bosh-deployment #[ex. deployments/bosh] +genesis info [env] +``` +`[env]` - name of the env/yaml file + +You should see that env `url` `username` `password` and `ca certificate`. Best approach is to put them into the Vault vs plaintext in `ci.yml`.\ +By convention they should be stored under something similar to: +```bash +secret/[env]/bosh/ssl/ca:certificate +secret/[env]/bosh/users/[user]:password +``` +Make sure crendtials are there before applying! + +#### Retrieve **vault** configuration +To configure Vault access and: +- get vault url +- get vault role and secret +``` +cd my-vault-deployment #[ex. deployments/vault] +genesis info [env] +``` +`[env]` - name of the env/yaml file + +If you don't have Vault installed via kit, try to get that information using `safe` CLI or `vault` CLI directly: +```bash +safe targets # shows all targets and `url` +safe status # shows all servers under current target +safe env # will get you url and `VAULT_TOKEN` +``` + +Once you get `VAULT_TOKEN` you can configure `approle` in it via API / `vault` CLI.\ +It is far more convenient to use the kit, but there is existing documentation from Vault how to proceed from that point: +https://www.vaultproject.io/docs/auth/approle From 196dba602819e266d5d493838359e7efb70e42c6 Mon Sep 17 00:00:00 2001 From: Benjamin Gandon Date: Tue, 8 Nov 2022 16:46:43 +0100 Subject: [PATCH 2/2] Suggest improvements in wording and presenting --- docs/PIPELINES.md | 97 +++++++++++++++++++++++++++++++---------------- 1 file changed, 64 insertions(+), 33 deletions(-) diff --git a/docs/PIPELINES.md b/docs/PIPELINES.md index eb917017..55e9acff 100644 --- a/docs/PIPELINES.md +++ b/docs/PIPELINES.md @@ -326,7 +326,7 @@ pipeline: #### Secrets - **pipeline.vault.url** - The URL of your Vault installation, i.e. - `https://vault.example.com` or `https://10.4.1.7`. This is **required**. + `https://vault.example.com` or `https://10.0.0.5`. This is **required**. - **pipeline.vault.role** - The AppRole GUID of a given Vault AppRole, used to generate temporary tokens for Vault-accessing during deploys. This field is @@ -563,70 +563,101 @@ being deployed, or vice versa. #### Pipeline Override Configuration If you need to edit and make changes to your deployment pipeline, simply add -the changes you need to the bottom of your ci.yml located in your -*-deployments repo to be merged by spruce. -For example, if we have a pipeline we need to edit the -'check_every:' parameter of a resource named git we can add the block below -under our pipeline: +the changes you need to the bottom of your `ci.yml` located in your +`*-deployments` repo to be merged by Spruce. + +For example, if we have a pipeline in which we need to enforce some +`check_every:` parameter on a resource named `git`, we can add the following +`resources:` block below under the pipeline generation settings specified in the +`pipeline:` section. **ci.yml** ``` - (...) - layouts: - test-sandbox-ops: |+ - auto *test-sandbox *test-staging - test-sandbox -> test-staging +pipeline: - groups: - default: - - test-sandbox - - test-staging + # (...) resources: -- name: git - check_every: 15m + - name: git # <-- required by default Spruce array merging, based on `name:` keys + check_every: 15m ``` +Spruce will merge anything outside the `pipeline:` section on top of the +generated Concourse pipeline definition. + ### Useful commands -#### Retrieve **bosh** configuration -In order to fill out the `pipeline.boshes.*` section first -we need to know all of the informations. -- get bosh url -- get bosh username and password -- get bosh cert\ -All of those params are available under +#### Retrieve **Bosh** configuration + +With recent Genesis versions, the Bosh URL and credentials are now grabbed from +the exodus data. + +Furthermore, even in situations where the Bosh password and CA certificate are +required, they should come from the Concourse integrated Vault. Typical path +would be `concourse///` and default value +for secrets is to be set under a `value:` key in Vault, as advised +[in the Concourse documentation][vault_creds_lookup_rules]. + +[vault_creds_lookup_rules]: https://concourse-ci.org/vault-credential-manager.html#vault-credential-lookup-rules + +In other situations where tests needs to be made, Spruce `(( vault ))` +directives can be used. In order to fill out the `pipeline.boshes.*` section +first we need to know all of the informations. + +- get Bosh URL +- get Bosh username and password +- get Bosh CA certificate + +All of those params are available under: + ```bash cd my-bosh-deployment #[ex. deployments/bosh] genesis info [env] ``` -`[env]` - name of the env/yaml file -You should see that env `url` `username` `password` and `ca certificate`. Best approach is to put them into the Vault vs plaintext in `ci.yml`.\ -By convention they should be stored under something similar to: +Where `[env]` is the name of the env/yaml file. + +You should see that env `url`, `username`, `password` and `ca certificate`. Best +approach is to put them into the Vault vs plaintext in `ci.yml`. + +By convention, they should be stored under something similar to: + ```bash secret/[env]/bosh/ssl/ca:certificate secret/[env]/bosh/users/[user]:password ``` -Make sure crendtials are there before applying! -#### Retrieve **vault** configuration +Make sure credentials are there before applying! + +#### Retrieve **Vault** configuration + +With recent Genesis versions, the genesis-pipeline app role should be +autodiscovered by the pipeline. + To configure Vault access and: + - get vault url - get vault role and secret + ``` cd my-vault-deployment #[ex. deployments/vault] genesis info [env] ``` -`[env]` - name of the env/yaml file -If you don't have Vault installed via kit, try to get that information using `safe` CLI or `vault` CLI directly: +Where `[env]` is the name of the env/yaml file. + +If you don't have Vault installed via kit, try to get that information using +`safe` CLI or `vault` CLI directly: + ```bash safe targets # shows all targets and `url` safe status # shows all servers under current target safe env # will get you url and `VAULT_TOKEN` ``` -Once you get `VAULT_TOKEN` you can configure `approle` in it via API / `vault` CLI.\ -It is far more convenient to use the kit, but there is existing documentation from Vault how to proceed from that point: +Once you get `VAULT_TOKEN` you can configure `approle` in it via API or `vault` +CLI. + +It is far more convenient to use the kit, but there is existing documentation +from Vault how to proceed from that point: https://www.vaultproject.io/docs/auth/approle