-
Notifications
You must be signed in to change notification settings - Fork 131
AP-2520 Add Iceberg table support to target-snowflake #1254
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
|
Comment
You can also request review from a specific team by commenting 💡 If you see something that doesn't look right, check the configuration guide. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds minimal support for Iceberg tables in Snowflake to PipelineWise. The implementation allows PipelineWise to work with pre-existing Iceberg tables without attempting to convert them back to native Snowflake tables. Key changes include type mapping adjustments (using string instead of variant for object/array types), detection logic to identify Iceberg tables, and safeguards to prevent automatic schema alterations on Iceberg tables.
Key Changes
- Added
is_iceberg_tableparameter tocolumn_type()function to handle Iceberg-specific type mappings - Implemented
iceberg_get_tables()method to query Snowflake metadata for Iceberg tables - Modified
update_columns()andsync_table()to detect Iceberg tables and prevent automatic alterations, requiring manual column changes instead
Reviewed Changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 9 comments.
| File | Description |
|---|---|
| singer-connectors/target-snowflake/target_snowflake/db_sync.py | Core implementation: added Iceberg table detection, type mapping for Iceberg compatibility, and logic to prevent automatic schema changes on Iceberg tables |
| singer-connectors/target-snowflake/target_snowflake/init.py | Updated log message for clarity ("PipelineWise table cache") |
| dev-project/docker-compose.yml | Adjusted Snowflake PEM file mount path from /opt/pipelinewise/.ssh/ to /root/.ssh/ |
| .gitignore | Added dev-project/snowflake.pem to ignored files for security |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This is currently a non-blocking check, and it will not prevent your Pull Request from being merged.The CODEOWNER of this repository has some untagged assets.As this is a public repository, asset details will not be shown. |
24d17f7 to
b701b26
Compare
Context
Add minimal support for PipelineWise to be able to write to an Iceberg table already defined in Snowflake.
PipelineWise will not create the Iceberg table, but if the table is already there it will not attempt to convert it back to a native table.
Types of changes
What types of changes does your code introduce to PipelineWise?
Put an
xin the boxes that applyChecklist
AP-NNN(if applicable. AP-NNN = JIRA ID)