feat(duckdb): add microbatch support for incremental strategies #644
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…d merge to the valid incremental strategies for dbt
…constants for strategy selection
…ew three configuration options
Added version-aware support for MERGE statements in DuckDB adapter for versions 1.4.0 and above. Introduced `_supports_merge` property to check and cache version compatibility, enabling "merge" and "microbatch" incremental strategies alongside existing "append" and "delete+insert". Updated SQL macros to implement microbatch and merge logic for efficient batch processing and upserts.
|
Oh wow, thanks for the tip! Will join the Slack to deduplicate efforts :) |
…port Merged changes from PR duckdb#605 (feature/merge-into-incremental-strategy) which adds comprehensive MERGE incremental strategy support with DuckDB extensions, while preserving microbatch incremental strategy support. Key changes: - Add comprehensive MERGE strategy with validation and custom clauses - Support for DuckDB MERGE extensions (BY NAME, BY POSITION, etc.) - Extensive configuration options for merge behavior - DuckLake restrictions validation - Preserve microbatch incremental strategy support - Reorganize macros into incremental_strategy/ directory - Add comprehensive merge strategy documentation to README 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
…detection - Move microbatch macro from delete_insert.sql to dedicated microbatch.sql - Microbatch now uses MERGE when available (DuckDB >= 1.4.0) for better performance - Falls back to delete+insert on earlier DuckDB versions - Follows incremental_strategy directory organization pattern 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
gfrmin
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added version-aware support for MERGE statements in DuckDB adapter for versions 1.4.0 and above. Introduced
_supports_mergeproperty to check and cache version compatibility, enabling "merge" and "microbatch" incremental strategies alongside existing "append" and "delete+insert". Updated SQL macros to implement microbatch and merge logic for efficient batch processing and upserts.