Added entombment of secrets

[*Major Security Improvement*]

* This release introduces 'entombment' of Vault secrets into Credhub for
  increased security in the transport of your secrets on deployment.

  In previous versions, the `(( vault ... ))` operators in the manifest
  fragments, your environment and ops/ YAML files would resolve locally,
  placing the secrets needed from vault into the manifest.
  Unfortunately, this would be done in clear text.

  The alternative secret store is Credhub, and a few kits still use
  Credhub to contain the deployments secrets.  However, Credhub lacks
  the ability to manage your secrets as well as Vault, requiring a
  different tooling, or just simply deleting all secrets and allowing
  them to be generated again.  The final issue is that becase the
  placeholder variable doesn't change, there is no visual confirmation
  that you have changed the value of a Credhub secret on deployment.

  Genesis 2.9.0 brings you the best of both worlds!

  Vault remains the single source of truth, and you can manage your
  secrets with the Genesis add-secrets, check-secrets, rotate-secrets
  and remove-secrets commands, independent of the BOSH director.

  However, when it comes time to deploy your manifest, Genesis
  determines what Vault paths are being used by the manifest, pulls them
  into memory, runs a sha1sum on the values, then determines Credhub
  path to inject each of those Vault values into based on its vault path
  and value.  This results in a unique and idempotent variable that only
  changes if the value it contains changes.  The value is stored at that
  path in Credhub if it doesn't already exist.

  When merging the manifest, the vault operators now resolve to that
  unique Credhub path that contains the desired value, and there are no
  longer any cleartext secrets in the manifest, the log, or even if you
  pull down the manifest using the `bosh manifest` command.

  This is enabled on default for the `genesis deploy` command, and can
  be enabled using the `--entomb` option for the manifest and lookup
  commands.  Likewise, it can be disabled on the deploy command with
  --no-entomb option.  You can also set it as a local setting in your
  ~/.genesis/config file with `entomb_secrets` key that can be set true
  or false.  The command line options will take precedent over the
  config file.

  Note: The values are left in Credhub after the deploy to ensure any
  recreate or other BOSH operations that require the manifest to be
  intact still work.  Over time, this will result in Credhub containing
  values that are no longer used, including expired X509 certificates,
  which may trigger false positives in Doomsday cert expiry monitoring
  tool.  Future version of Genesis will enable cleaning out any old
  values that are no longer being used.

Author: dennisjbell

bin/genesis

@@ -312,6 +312,11 @@ define_command("manifest", {
 			"Determines if vault values are fetched or redacted.  By default, the ".
 			"manifest will be redacted unless being output to the live console.",
 
+		'entomb' =>
+			"Will entomb valut secrets into the BOSH director's credhub as is done ".
+			"when deploying, providing a manifest that does not contain any clear ".
+			"text secrets",
+
 		'prune!' =>
 			"Determines if the build metadata is pruned.  Defaults to true.",
 
@@ -384,6 +389,14 @@ define_command("deploy", {
 
 		'max-in-flight=i' =>
 			"Override the default number of maximum VMs in flight per instance group.",
+
+		'entomb!' =>
+			"By default, deployments to BOSH directors will entomb vault secrets into ".
+			"the director's CredHub, then use CredHub variable operators to prevent ".
+			"secrets from being present in clear text in the manifest.  This can be ".
+			"prevented by specifying --no-entomb as an option, or by setting ".
+			"'entomb_secrets: false' in your #C{~/.genesis/config} file.  The option ".
+			"will override the configuration file setting.",
 	]
 });
 # }}}
@@ -450,7 +463,7 @@ define_command("info", {
 # genesis lookup - Find a key set in environment manifests. {{{
 define_command("lookup", {
 	summary => "Look up values from an environment's file, manifest, deployment, exodus or env state.",
-	usage => "<env> lookup [--merged|--deployed|--env|--exodus|] key [default-value|--defined]\n".
+	usage => "<env> lookup [--merged|--deployed|--env|--exodus|] key [default-value|--defined] [--entomb]\n".
 					 "lookup --exodus-for <env/type> key [default-value|--defined]",
 	alias => 'get',
 	description =>
@@ -474,6 +487,11 @@ define_command("lookup", {
 			"Takes an argument of the form 'env-name/deployment-type'",
 		"env" =>
 			"Lookup environment variables used by Genesis for the given environment.",
+		"entomb" =>
+			"Entomb the vault secrets in the BOSH director's credhub before ".
+			"rendering the manifest, so that if looking up a secret, return the ".
+			"reference to credhub instead.  Not applicable to exodus or deployed ".
+			"manifests",
 		"defined" =>
 			"Exit with 0 if key defined in specified source, 9 otherwise.  No output ".
 			"is produced, making it useful in 'if lookup ... ; then'"

lib/Genesis/Commands/Env.pm

@@ -254,24 +254,26 @@ sub manifest {
 		);
 	}
 
-	print $env
+	output {raw => 1}, $env
 		->download_required_configs('blueprint', 'manifest')
 		->manifest(
 			partial   => get_options->{partial},
 			redact    => get_options->{redact},
 			prune     => get_options->{prune},
-			vars_only => get_options->{'bosh-vars'}
+			vars_only => get_options->{'bosh-vars'},
+			entomb    => get_options->{entomb}
 		);
 }
 
 sub deploy {
 	option_defaults(
-		redact => ! -t STDOUT
+		redact => ! -t STDOUT,
+		entomb => $Genesis::RC->get('entomb_secrets',1)
 	);
 	command_usage(1) if @_ != 1;
 
 	my %options = %{get_options()};
-	my @invalid_create_env_opts = grep {$options{$_}} (qw/fix dry-run/);
+	my @invalid_create_env_opts = grep {$options{$_}} (qw/fix dry-run entomb/);
 
 	$options{'disable-reactions'} = ! delete($options{reactions});
 	my $env = Genesis::Top->new('.')->load_env($_[0])->with_vault();
@@ -286,6 +288,8 @@ sub deploy {
 		join(", ", @invalid_create_env_opts)
 	) if $env->use_create_env && @invalid_create_env_opts;
 
+	$options{entomb} = 1 unless defined($options{entomb}) || $env->use_create_env;
+
 	info "Preparing to deploy #C{%s}:\n  - based on kit #c{%s}\n  - using Genesis #c{%s}", $env->name, $env->kit->id, $Genesis::VERSION;
 	if ($env->use_create_env) {
 		info "  - as a #M{create-env} deployment\n";

lib/Genesis/Commands/Info.pm

@@ -133,6 +133,13 @@ sub lookup {
 	}
 	my $env = $top->load_env($name);
 	my $v;
+	if (get_options->{entomb}) {
+		bail(
+			"Cannot use --entombed option with --exodus, --exodus-for or --deployed",
+		) if scalar( grep {$_} (@{get_options()}{qw/exodus exodus-for deployed/}));
+		$env->entombed_secrets_enabled(1);
+	}
+
 	if (get_options->{merged}) {
 		bail(
 			"Circular reference detected while trying to lookup merged manifest of $name"
@@ -167,7 +174,7 @@ sub lookup {
 		exit(ref($v) eq "NotFound" ? 4 : 0);
 	} elsif (defined($v)) {
 		$v = encode_json($v) if ref($v);
-		output {raw => 1}, "$v\n";
+		output {raw => 1}, $v;
 	}
 	exit 0;
 }