The datatovalue-tools library extends Google BigQuery to simplify common Analytics Engineering use-cases which we encounter daily working with clients at Data to Value.
More specifically, the library comprises a set of utility functions to support data profiling, debugging, root-cause analysis and automation activities in data transformation and management. It is developed and maintained by the Engineering Team at Data to Value and is licensed under Apache 2.0.
If you are interested in making a contribution or suggestion, please contact [email protected].
The following sets of functions are deployed in the datatovalue-tools Google Cloud project, across all available regions.
| Function Set | Description |
|---|---|
| Metadata Functions | Functions supporting metadata-based analysis and automation. |
| Monitor Functions | Functions which are used to monitor BigQuery resources and resource status. |
| Profile Functions | Functions to support data quality assurance activities by modelling, integrating and analysing table contents and metadata. |
| JSON Functions | Functions to support automatic schema parsing and JSON parser deployment, to support accurate data transfer and efficient data pipeline development. |
| SQL Parsers | Functions which parse SQL queries and derive the logical structure and dependency graph. |
Functions are live and deployed across in the datatovalue-tools BigQuery project for all global regions and multi-regions. Note that the dataset names contain underscores instead of dashes (e.g. functions in europe-north1 are in the dataset datatovalue-tools.europe_north1).
Functions can be called by any user with BigQuery Data Viewer, BigQuery Data User or higher permissions on your source data. All authenticated users are permitted to call functions in the datatovalue-tools regional datasets.