Skip to content

Conversation

@louis-pie
Copy link
Contributor

Context

Add minimal support for PipelineWise to be able to write to an Iceberg table already defined in Snowflake.
PipelineWise will not create the Iceberg table, but if the table is already there it will not attempt to convert it back to a native table.

Types of changes

What types of changes does your code introduce to PipelineWise?
Put an x in the boxes that apply

  • Bugfix (non-breaking change which fixes an issue) [optional]
  • New feature (non-breaking change which adds functionality) [optional]
  • Breaking change (fix or feature that would cause existing functionality to not work as expected) [optional]
  • Documentation Update (if none of the other choices apply) [optional]

Checklist

  • I have read the CONTRIBUTING doc
  • Description above provides context of the change
  • I have added tests that prove my fix is effective or that my feature works
  • Unit tests for changes (not needed for documentation changes)
  • CI checks pass with my changes
  • Branch name starts with AP-NNN (if applicable. AP-NNN = JIRA ID)
  • Relevant documentation is updated including usage instructions

Copilot AI review requested due to automatic review settings November 21, 2025 08:49
@louis-pie louis-pie requested a review from a team as a code owner November 21, 2025 08:49
@platon-github-app-production
Copy link

Comment /request-review to automatically request reviews from the following teams:

You can also request review from a specific team by commenting /request-review team-name, or you can add a description with --notes "<message>"

💡 If you see something that doesn't look right, check the configuration guide.

@louis-pie louis-pie changed the title AP-2520 target snowflake iceberg AP-2520 Add Iceberg table support to target-snowflake Nov 21, 2025
Copilot finished reviewing on behalf of louis-pie November 21, 2025 08:53
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds minimal support for Iceberg tables in Snowflake to PipelineWise. The implementation allows PipelineWise to work with pre-existing Iceberg tables without attempting to convert them back to native Snowflake tables. Key changes include type mapping adjustments (using string instead of variant for object/array types), detection logic to identify Iceberg tables, and safeguards to prevent automatic schema alterations on Iceberg tables.

Key Changes

  • Added is_iceberg_table parameter to column_type() function to handle Iceberg-specific type mappings
  • Implemented iceberg_get_tables() method to query Snowflake metadata for Iceberg tables
  • Modified update_columns() and sync_table() to detect Iceberg tables and prevent automatic alterations, requiring manual column changes instead

Reviewed Changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 9 comments.

File Description
singer-connectors/target-snowflake/target_snowflake/db_sync.py Core implementation: added Iceberg table detection, type mapping for Iceberg compatibility, and logic to prevent automatic schema changes on Iceberg tables
singer-connectors/target-snowflake/target_snowflake/init.py Updated log message for clarity ("PipelineWise table cache")
dev-project/docker-compose.yml Adjusted Snowflake PEM file mount path from /opt/pipelinewise/.ssh/ to /root/.ssh/
.gitignore Added dev-project/snowflake.pem to ignored files for security

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@wise-github-bot-app
Copy link

This is currently a non-blocking check, and it will not prevent your Pull Request from being merged.

The CODEOWNER of this repository has some untagged assets.

As this is a public repository, asset details will not be shown.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants