dbtwiz

Python package with CLI helper tool for dbt in GCP using BigQuery. Although some functions are platform independent, the majority assume GCP and BigQuery is used.

Installation

pip install git+https://github.com/amedia/dbtwiz

Subcommands

These are the available subcommands for dbtwiz. You can also run dbtwiz --help/dbtwiz -h, which will list the commands with more details.

• model - Commands for a dbt model
  • create - Create new dbt model.
  • fix - Run sqlfmt and sqlfix for staged and/or defined sql files.
  • from-sql - Convert a sql file to a dbt model by replacing table references with source and ref.
  • inspect - Output information about a given model.
  • lint - Run sqlfmt --diff and sqlfluff lint for staged and/or defined sql files.
  • move - Moves a model by copying to a new location with a new name, and/or by updating the references to the model by other dbt models.
• source - Commands for a dbt source
  • create - Create new dbt source
• build - Build one or more dbt models, using interactive selection with fuzzy-matching, unless an exact model name is passed.
• test - Test dbt models
• manifest - Update dev and production manifests for fast lookup
• backfill - The backfill subcommand allows you to (re)run date-partitioned models in production for a period spanning one or multiple days. It will spawn a Cloud Run job that will run dbt for a configurable number of days in parallel.
• config - Update user configuration setting
• admin - Administrative commands
  • cleandev - Delete all materializations in the dbt development dataset
  • orphaned - List or delete orphaned materializations in the data warehouse
  • partition-expiry - Checks for mismatched partition expiry and allows updating to correct.

Configuration

Project config

Depending on the specific subcommand, there are some configuration settings defined in a pyproject.toml file that the tool will look for.

The tool will give you a warning when you run a commmand that needs one of the config elements should it be missing, so you don't need to add them all before they become relevant.

[tool.dbtwiz.project]
# Config for bucket containing dbt manifest.json at the top level
bucket_state_project = ""         # Project name for bucket
bucket_state_identifier = ""      # Bucket name

# Config for service account used for backfill and cleanup of orphaned models in prod
service_account_project = ""      # Project name for where service account actions are run
service_account_identifier = ""   # Name of service account
service_account_region = ""       # Region for where service account actions are run

# Config for user actions
user_project = ""                 # Project name for where user queries are run
user_auth_verified_domains = []   # Which domains to check when identifying whether user is already authenticated

# Config for docker image used for backfill
docker_image_url_dbt = ""         # Url for docker image
docker_image_profiles_path = ""   # Path to profiles dir in docker image
docker_image_manifest_path = ""   # Path to manifest in docker image

User config

The default configuration of dbtwiz will be installed the first time you run it, but you may want to adjust some settings from the get-go to fit your environment.

Dark mode

If you're using a dark background colour in your terminal, you should configure dbtwiz to use bright colours for highlighting in previews and elsewhere to make the text more readable.

Run the following command to switch from default light mode to dark mode:

$ dbtwiz config theme dark

Preview formatter

By default, dbtwiz uses the command fmt tool to format text in the preview window when selecting models interactively. Under macOS, the fmt tool won't handle ANSI escape codes, and unless you have the GNU coreutils version of fmt you will get garbage characters in the preview window, and should switch to the simple cat command for formatting instead:

$ dbtwiz config model_info:formatter "cat -s"

Development

# where you keep locally checked out repos:
git clone [email protected]:amedia/dbtwiz

# inside the virtual environment of your dbt project:
pip install -e <local-path-to-dbtwiz-repository>