Wayfinder CloudResourceDataSources

This repo contains the types of CloudResource dependency that wayfinder supports.

Overview

CloudResourceDataSources provide a way for Wayfinder to discover resource dependencies for developers.

The resources are typically provided by an organization's cloud estate and will be discovered using tags.

Each resource kind can be referenced as a dependency of another cloud resource being built and discovered at run time.

CloudResourceDataSources define terraform with configuration that is used for read only discovery not creating resources.

Highlevel Orchestration Workflow

1. A CloudResourceSearch is created by the system during a deployment when developer needs to know about dependencies. The tag values could have developer relevant metadata.
2. A CloudResourceDataSource is used by the CloudResourceSearch controller and the cloud resource search is run.
3. Identifiers are shown to a developer to resolve ambigouse search results.
4. A concrete single cloud resource object is created using a CloudResourceDataSource reference using the singular terraform to resolve the detailed outputs.

Terraform in CloudResourceDataSource - Limitations and Rules

The CloudResourceDataSource terraform is orchetsrated and run in a constrained, pre-configured runtime environment for both searches and resource audit:

• Only defines data sources and outputs (read only).
• Has a pre-configured default provider configuration for each cloud.
• Will have only ready only access to cloud and is used for auditing cloud resources only.
• Can only use additional providers using the API and the spec.terraform.additionalProviders array field.

Searches

• Uses the variable var.resource_tags to provide the search input with developer instance relevant data.
• Uses the output output to store an array of maps with identifier values.

Resource

• Uses golang templating to create a set of terraform variables named using a single array item from the output array of map identifiers from above
• Provides the outputs as defined to other dependent resources.

New DataSource Tests

Test Files

BEFORE we can test finding resources using a CloudResourceSearch and CloudResourceDataSource we need to have the resources to find!

Files to create an instance of a resource (NOT discover).

1. Create a CloudResourcePlan .yaml file in tests/[datasource]/[datasource]-cr-plan.yaml that will define how to create an instance of the kind.

   This must be to a valid terraform module to create (not find the resource). This will have valid tags so that we can find the resource later.

   It is imperative that the plan will work with the terraform module referenced and all required variables are templated suitably.

2. Optionally create a create.env file to set any variables used by the wf deploy -f [datasource]-wayfinder-create.yaml .... E.g. Get some dependant fields used in creation:

   WF_VAR_rgName="$(cat ../azurerm-resource-group/out.json | jq -r '.componentOutputs.rg.name.value')"
   

3. Create a Wayfinder deployment yaml file in tests/[datasource]]/[datasource]-wayfinder-create.yaml this will create the instance of the plan.

   wf deploy is used with this file to create the test resource against a valid real cloud target.

4. Create a search.env file in tests/[datasource]]/search.env to set the test parameters. Use FILTER to define how to carry out the search (given resources created):

   FILTER="vpc_id=$(cat ../aws-vpc/out.json | jq -r '.componentOutputs.vpc.vpc_id.value')"
   

   Optionally use the REQUIRES_DATASOURCE_CREATE to specify a dependency of the test:

   REQUIRES_DATASOURCE_CREATE="aws-vpc"