Add MERGE incremental strategy with DuckDB extensions support #605

VahOkan · 2025-08-05T19:43:45Z

Implements MERGE incremental strategy for dbt-duckdb with support for DuckDB's advanced
MERGE extensions from PR #18135.

Features

Standard MERGE functionality
DuckDB extensions:
- UPDATE/INSERT BY NAME, UPDATE/INSERT BY POSITION, UPDATE SET *
- WHEN NOT MATCHED BY SOURCE
- DO NOTHING and ERROR actions
- USING clause with explicit columns
Custom merge clauses with multiple actions, conditions, and error handling
Version checking (requires DuckDB >= 1.4.0)
Comprehensive test suite covering basic, enhanced, and custom configurations

Configurations Added

Basic config:
- unique_key
Enhanced config:
- incremental_predicates
- merge_on_using_columns
- merge_update_condition
- merge_insert_condition
- merge_update_columns
- merge_exclude_columns
- merge_update_set_expressions
- merge_returning_columns
Custom config:
- merge_clauses with support for when_matched and when_not_matched actions/modes
Supported actions: update, insert, delete, do_nothing, error

Notes

MERGE requires DuckDB >= 1.4.0
When using DuckLake, only a single UPDATE or DELETE action is allowed in when_matched clauses due to current limitations.

…d merge to the valid incremental strategies for dbt

…ategy folder

…tions

…ation params

jwills

happy to just keep running these tests as you push new changes in; the linting stuff is done via pre-commit run --all-files IIRC

jwills · 2025-08-06T20:46:16Z

dbt/adapters/duckdb/impl.py

+            _, cursor = self.connections.add_select_query("SELECT version()")
+            version_string = cursor.fetchone()[0]
+
+            logger.info(f"DuckDB version: '{version_string}'")


FYI that I'm going to want these logger lines to go or be downgraded to debug prior to merging

@jwills Thanks for the hints. I removed some of the logger lines that were meant for debugging, ran pre-commit and tox -p auto, and pushed again.

I checked the logs of some of the tests — they are failing because tox is not using DuckDB 1.4.0, right? I ran the tests locally after updating tox to use the DuckDB preview version 1.4.0.dev2079 with .tox/functional/bin/pip install --pre duckdb --upgrade

ah duh of course-- I'll have to try them out locally

VahOkan · 2025-08-08T15:45:46Z

FYI: @jwills I just added a validation for the DuckLake MERGE restrictions.

jwills

looking really good-- couple of things i'd like to fix and one more of a style thing

jwills · 2025-08-08T17:07:14Z

dbt/adapters/duckdb/impl.py

+        """DuckDB supports MERGE statements starting from version 1.4.0."""
+        base_strategies = ["append", "delete+insert"]
+
+        if self._supports_merge_strategy():


I'm not wild about doing this on every call to valid_incremental_strategies()-- i'd rather the set of valid strategies be cached s.t. this is only done once per dbt run instead of every time we need to check for this

I’ve updated the code to use cached properties for the DuckDB version and incremental strategy values, and I’m now using the Version class from packaging to compare versions. This is a more Pythonic and elegant approach.

jwills · 2025-08-08T17:07:29Z

dbt/adapters/duckdb/impl.py

+        if self._supports_merge_strategy():
+            return base_strategies + ["merge"]
+
+        logger.info(


Another logger line that I would like to be downgraded to debug and/or removed

I removed the logger line but added an exception in case the user tries to use merge with an older version. I had to override the get_incremental_strategy_macro for this; otherwise, the user would get an exception stating that merge is not available for the adapter, which is no longer true after this PR.

jwills · 2025-08-08T17:09:27Z

dbt/include/duckdb/macros/materializations/incremental_strategy/validations.sql

@@ -0,0 +1,236 @@
+{% macro validate_merge_update_options(update_all, update_by_name, update_by_position) %}


Mostly a style/vibe comment here-- this is a lot of validation code entirely in jinja; is this how other adapters that have complex merge logic handle it? I'm inclined to keep it b/c I think moving stuff to jinja is generally a good thing as we look towards a dbt-fusion future, just something I wanted to call out in case there were better pattern for this

None of the adapters' merge logic is as complex as this one. The ones for BigQuery and Snowflake just use the default get_merge_sql from dbt-core with some additional code. And yes, usually they just use Jinja for this kind of stuff.

DuckDB introduced many additional extensions to the basic merge and i am trying to take advantage of these extensions, which is why the code is more complex in our case. Thus we also need more validations to ensure that people configure them correctly. In other adapters, I couldn't find much validation code in incremental strategies because they simply don't have a lot of configurations to validate.

That said, I still agree with you that some of the code could be removed to improve performance and readability. I could redesign it in the following way:

More compact configurations — e.g., merge_update_by expecting a value 'position' or 'name' instead of having two booleans. Still, I would need to validate if the user provides the correct options.

Removing some basic configuration validations by trusting that people won’t mess them up.

Only one merge validation macro with validation rules and a loop, instead of having many separate macros.

What do you think @jwills?

Sounds good to me-- thanks!

Please check the README.md first — I think it’s the easiest way to understand my approach. In short: it requires only minimal configuration for basic use cases (about 90% I’d guess) while still providing maximum flexibility for more complex scenarios by taking advantage of DuckDB’s MERGE syntax extensions.

In terms of validations, I used a more modular approach this time, which reduced the overall code. Most of the validation logic isn’t even used for basic configurations, so performance shouldn’t be an issue. For complex use cases, the few extra milliseconds added to execution time shouldn’t bother anyone.

Also added a validation to check for DuckLake based on the restrictions from duckdb/ducklake#351

…constants for strategy selection

…re user friendly

…ew three configuration options

jwills · 2025-09-17T14:41:20Z

@VahOkan ahh now maybe ducklake 0.3 does support MERGE?

VahOkan · 2025-09-18T09:58:29Z

@VahOkan ahh now maybe ducklake 0.3 does support MERGE?

I think MERGE is supported in Ducklake, but with some restrictions. I’ve tried to account for those restrictions in my PR, but it should be tested to be sure.

jwills

@VahOkan sorry for the insane lag here, just a hectic month at work. Tripped over some stuff with the merge logic that I think we need to fix up + have test coverage for before we can merge it in.

jwills · 2025-10-06T20:43:59Z

dbt/include/duckdb/macros/materializations/incremental_strategy/merge.sql

+                {%- set exclude_columns = when_matched.get('update', {}).get('exclude', []) -%}
+                {%- set set_expressions = when_matched.get('update', {}).get('set_expressions', {}) -%}
+
+                {%- set update_columns = get_merge_update_columns(include_columns, exclude_columns, dest_columns) -%}


Wheres does this method get defined? Did I lose it somehow in my rebases or am I just missing it?

jwills · 2025-10-06T20:45:37Z

dbt/include/duckdb/macros/materializations/incremental_strategy/merge.sql

+            {%- elif when_not_matched.get('mode') == 'star' -%}
+            INSERT *
+            {%- elif when_not_matched.get('mode') == 'explicit' -%}
+            {%- set insert_columns = when_matched.get('insert', {}).get('columns', []) -%}


Are these lines supposed to reference when_not_matched? I don't totally follow the logic here

jwills · 2025-10-06T20:47:05Z

dbt/include/duckdb/macros/materializations/incremental_strategy/merge.sql

+    MERGE INTO {{ target }} AS DBT_INTERNAL_DEST
+        USING {{ source }} AS DBT_INTERNAL_SOURCE
+        {%- if merge_on_using_columns %}
+            USING ({{ merge_on_using_columns | join(', ') }})


Is this supposed to be ON ({{ merge_on_using_columns | join(', ') }})? I don't grok the USING ... USING ... syntax

…port Merged changes from PR duckdb#605 (feature/merge-into-incremental-strategy) which adds comprehensive MERGE incremental strategy support with DuckDB extensions, while preserving microbatch incremental strategy support. Key changes: - Add comprehensive MERGE strategy with validation and custom clauses - Support for DuckDB MERGE extensions (BY NAME, BY POSITION, etc.) - Extensive configuration options for merge behavior - DuckLake restrictions validation - Preserve microbatch incremental strategy support - Reorganize macros into incremental_strategy/ directory - Add comprehensive merge strategy documentation to README 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Vahe Hakobyan added 10 commits August 5, 2025 21:25

check if duckdb version supports the merge strategy (>= 1.4.0) and ad…

3496eb8

…d merge to the valid incremental strategies for dbt

remove the incremental_helper to move the code to the incremental_str…

b25f3fa

…ategy folder

merge macros for duckdb with additional duckdb merge into extensions

63adac2

add tests for the merge incremental strategy with different configura…

cebe3fa

…tions

add tests for the merge with update and exclude columns configurations

245b441

put the delete+insert merge code do a seperated module

d17013f

validation module with merge validation macros for the merge configur…

49c5deb

…ation params

add the validations to the merge macro + some improvements

cfcefa4

new test to check if the validations work

fa6f283

add documentation for the incremental strategies

1b20f1f

jwills reviewed Aug 6, 2025

View reviewed changes

Vahe Hakobyan added 5 commits August 7, 2025 00:07

improve the documentation for incremental strategy configurations

5371c23

pre-commit all files

bea8b40

remove the logger lines

cb5b56c

pre-commit

fdebb52

add validation for ducklake merge restrictions

ab05e0d

jwills reviewed Aug 8, 2025

View reviewed changes

Vahe Hakobyan and others added 11 commits August 9, 2025 00:15

add DuckDBVersion class for semantic version parsing and comparison

d11599b

add duckdb incremental strategy and merge version constants

5d7aa71

refactor incremental strategy logic to use cached duckdb version and …

8c12a25

…constants for strategy selection

use the packaging version class for the duckdb version property

081eb05

redesign the merge incremental strategy to make the configurations mo…

1bceec9

…re user friendly

default merge clauses for the basic merge configurations

241423d

redesign the merge validations making with more modular approach

c947840

new tests for the merge incremental strategy

9e1a8a6

Adjust the merge incremental strategy documentation to describe the n…

22e7d2c

…ew three configuration options

Merge branch 'master' into feature/merge-into-incremental-strategy

2c31f48

linting stuff

7eedb92

jwills reviewed Oct 6, 2025

View reviewed changes

jwills mentioned this pull request Oct 8, 2025

feat(duckdb): add microbatch support for incremental strategies #644

Open

		@@ -0,0 +1,236 @@
		{% macro validate_merge_update_options(update_all, update_by_name, update_by_position) %}

Add MERGE incremental strategy with DuckDB extensions support #605

Are you sure you want to change the base?

Add MERGE incremental strategy with DuckDB extensions support #605

Uh oh!

Conversation

VahOkan commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Features

Configurations Added

Notes

Uh oh!

jwills left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VahOkan Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VahOkan commented Aug 8, 2025

Uh oh!

jwills left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VahOkan Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VahOkan Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VahOkan Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VahOkan Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jwills commented Sep 17, 2025

Uh oh!

VahOkan commented Sep 18, 2025

Uh oh!

jwills left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

VahOkan commented Aug 5, 2025 •

edited

Loading

VahOkan Aug 6, 2025 •

edited

Loading

VahOkan Aug 11, 2025 •

edited

Loading

VahOkan Aug 11, 2025 •

edited

Loading

VahOkan Aug 8, 2025 •

edited

Loading

VahOkan Aug 11, 2025 •

edited

Loading