feat(cli): add contract fetch command

[adkinsty]

Summary

As discussed on slack here, I thought it would be nice to have a contract pull command in soda core. So, I am proposing these changes to add such a contract pull command. The command requires three arguments (-d, -c and -sc) as described here:

usage: soda contract pull [-h] [-d DATASET [DATASET ...]] -sc SODA_CLOUD [-c CONTRACT [CONTRACT ...]] [-v]

options:
  -h, --help            show this help message and exit
  -d DATASET [DATASET ...], --dataset DATASET [DATASET ...]
                        Fully qualified names of datasets whose cloud contracts you wish to fetch.
  -sc SODA_CLOUD, --soda-cloud SODA_CLOUD
                        A Soda Cloud configuration file path.
  -c CONTRACT [CONTRACT ...], --contract CONTRACT [CONTRACT ...]
                        One or more contract file paths.
  -v, --verbose         Show more detailed logs on the console.

The command fetches the contract YAML strings from Soda Cloud for a given dataset and then writes these strings to the given YAML file paths. The command re-uses the same soda_cloud_client.fetch_contract_for_dataset(dataset_identifier) method used by the contract verify command.

Example usage

soda contract pull -d demo_unity_catalog/unity_catalog/tyler/obs_test_data -sc sc.yml -c /Users/adkinsty/PycharmProjects/soda-core-contracts/obs_test_data.yml

Prints:

  __|  _ \|  \   \\
\__ \ (   |   | _ \\
____/\___/___/_/  _\\ CLI 4.0.0.dev??
📜 Fetching contract from Soda Cloud for dataset 'demo_unity_catalog/unity_catalog/tyler/obs_test_data'

And then a new file is created at /Users/adkinsty/PycharmProjects/soda-core-contracts/obs_test_data.yml with the following contents:

dataset: demo_unity_catalog/unity_catalog/tyler/obs_test_data
filter: timestamps >= current_timestamp() - INTERVAL 1 DAY
soda_agent:
  checks_schedule:
    cron: 0 8 * * *
    timezone: America/New_York
checks:
  - row_count:
      name: Table is not empty
  - freshness:
      column: timestamps
      name: Data is less than 48 hrs old
      threshold:
        unit: hour
        must_be_less_than: 48
  - schema:
      allow_other_column_order: true
      name: Schema is valid
  - duplicate:
      name: No duplicate records
      columns:
        - id
        - latencies
        - responses
        - timestamps
columns:
  - name: id
    data_type: STRING
    checks:
      - missing:
          name: ID is complete
      - duplicate:
          name: ID is unique
      - invalid:
          name: ID has valid format (uuid4)
          invalid_format:
            name: uuid4
            regex: \b[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}\b
  - name: latencies
    data_type: DOUBLE
    checks:
      - aggregate:
          function: min
          name: Response latency is positive
          threshold:
            must_be_greater_than_or_equal: 0
      - invalid:
          name: Response latencies are less than 10 seconds
          valid_max: 10
  - name: responses
    data_type: STRING
    checks:
      - invalid:
          valid_values:
            - f
            - j
          threshold:
            metric: count
            must_be: 0
          name: Responses are valid (f, j)
  - name: timestamps
    data_type: TIMESTAMP

[CLAassistant]

All committers have signed the CLA.

[nielsn]

Very good stuff! Was not entirely sure about the "pull" terminology (as we don't have a "push" either) so maybe "fetch" or "get"? Don't feel strongly though.

[sonarqubecloud]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[adkinsty]

@nielsn @m1n0 Thanks for reviewing. I addressed the changes suggested above.

• Renamed the pull command to fetch
• Renamed the --contract argument to --file
• Made arg definition multi-line (for more consistent style)
• Bonus: added handling for missing directories and improved logging (based on code for create datasource).

Example usage:

Command:

soda contract fetch -d demo_unity_catalog/unity_catalog/tyler/obs_test_data -sc sc.yml -f /Users/adkinsty/PycharmProjects/soda-core-contracts/test/obs_test_data.yml

Output:

  __|  _ \|  \   \\
\__ \ (   |   | _ \\
____/\___/___/_/  _\\ CLI 4.0.0.dev??
📜 Fetching contract from Soda Cloud for dataset 'demo_unity_catalog/unity_catalog/tyler/obs_test_data'
✅  Created contract source file '/Users/adkinsty/PycharmProjects/soda-core-contracts/test/obs_test_data.yml'

Command:

soda contract fetch -d demo_unity_catalog/unity_catalog/tyler/obs_test_data -sc sc.yml -f /Users/adkinsty/PycharmProjects/soda-core-contracts/test/obs_test_data.yml

Output:

  __|  _ \|  \   \\
\__ \ (   |   | _ \\
____/\___/___/_/  _\\ CLI 4.0.0.dev??
📜 Fetching contract from Soda Cloud for dataset 'demo_unity_catalog/unity_catalog/tyler/obs_test_data'
✅  Updated contract source file '/Users/adkinsty/PycharmProjects/soda-core-contracts/test/obs_test_data.yml'

[m1n0]

very good, thanks!

[adkinsty]

very good, thanks!

Thanks @m1n0 @nielsn.

Is this automatically deployed to the dev pypi?