ducklake integration

[YuweiXiao]

This PR is a proof of concept for integrating ducklake into pg_duckdb, named pg_ducklake.

A new table AM called ducklake is introduced and supports create/insert/query ducklake tables directly through Postgres.

CREATE TABLE t (id int) USING ducklake;
INSERT INTO t SELECT g % 10 from generate_series(1,1000) g;
SELECT COUNT(*) FROM t;

Some TODOs:

• Support update/delete
• Add GUC for base data path. Consider adding OID of database/schema/table in the data path.
• Verify DuckDB CLI can read tables generated by pg_duckdb.
• Partitioned table support. How PG partition tables map to DuckLake partitioned tables?
• Inline data tables.
• Vacuum for dead files, dropped tables, files from aborted transactions
• ...

[chris-gooch]

Will this have local iceberg table and/or columnar storage support?

[YuweiXiao]

Will this have local iceberg table and/or columnar storage support?

The integration introduces columnar storage support to pg_duckdb (ps. it is not Iceberg tables). However, the upstream DuckLake plans to support migrating existing Iceberg tables to DuckLake (duckdb/ducklake#36).

[kakoni]

Hi @YuweiXiao! Interesting patch, to make this bit more usable, could data_path be configurable(=now hardcoded to /tmp/ducklake)

[YuweiXiao]

Hi @YuweiXiao! Interesting patch, to make this bit more usable, could data_path be configurable(=now hardcoded to /tmp/ducklake)

Yeah sure. Plan to have a guc to control the path and persist it into table metadata during table creation.

[mikewhb]

@YuweiXiao Are you still actively working on this patch?

[YuweiXiao]

@YuweiXiao Are you still actively working on this patch?

Sorry, no. Given that DuckLake is currently in active development, so gonna wait till a stable release of it. In the meantime, would like to get more feedback on both the features and how it is implemented.

[mikewhb]

@YuweiXiao Are you still actively working on this patch?

Sorry, no. Given that DuckLake is currently in active development, so gonna wait till a stable release of it. In the meantime, would like to get more feedback on both the features and how it is implemented.

Thank you so much for taking the time to reply. 能建立个联系吗？ 先加个邮箱（126邮箱 mymike_cn ）再加微信