feat(datasets): make table write mode configurable #1093

gitgud5000 · 2025-05-31T01:33:54Z

TODO(deepyaman): Move notes on credentials support to a new PR from @gitgud5000 with those changes, once it exists. I'm not deleting the detailed notes from here until they're moved elsewhere.

feat(datasets): ibis.TableDataset add mode and credentials support

feat: Introduces a mode parameter for save operations, allowing "append", "overwrite", "error", and "ignore" options to control write behavior.
- Supports legacy overwrite flag with backward compatibility and deprecation warning.
- Adds mode dispatching logic to handle different write scenarios.
- Prevents simultaneous use of both mode and legacy overwrite to avoid ambiguity.
- Updates examples and docs to reflect new parameter.
feat: Adds a credentials parameter to accept connection info (dict or string URI), superseding connection.
- Raises a warning if both credentials and deprecated connection are provided.
- Adds backend extraction helper and adjusts _describe() to use it.
- Improves documentation to reflect the preferred usage.

Semi-breaking change: overwrite and connection are deprecated; users should migrate to mode and credentials.

Description

This PR introduces two key enhancements to ibis.TableDataset:

Configurable Save Modes:
- Adds a mode parameter to save_args similar to Spark’s DataFrameWriter.mode and Pandas’ to_csv(mode=...), with support for:
  - "append": Insert data into an existing table (requires the backend to implement insert()).
  - "overwrite": Drop and recreate the table/view.
  - "error" or "errorifexists": Fail if the table already exists.
  - "ignore": Do nothing if the table exists; otherwise, create it.
- Backward compatible: legacy overwrite=True|False maps to mode="overwrite"|"error".
- Raises an error if both mode and overwrite are specified simultaneously.
credentials Parameter:
- Introduces credentials as the preferred method for specifying Ibis backend connection configurations.
- Accepts:
  - A string connection URI.
  - A dictionary of parameters (with optional con string).
- Supersedes the older connection parameter.
- Warns if both are provided.
- Updates both _connect() and _describe() to support new functionality.

Development notes

Closes Support for inserting data in ibis.TableDataset #834 (requested by @deepyaman on Sep 14, 2024, to add “insert” support via a mode argument).
- In the parent issue Enhance current integration between Kedro and Ibis #1174, the main ask was to allow an “append”/“insert” operation for Ibis datasets.
- This implementation covers:
  - Rework the API akin to Spark’s DataFrameWriter.mode and Pandas’s to_csv(mode=…).
  - Backward compatibility with overwrite behavior.
  - Clear, explicit behavior for each mode option.
  - Support for passing credentials in different formats supported by ibis.connect()

Expand for details:

Save Mode Handling

DEFAULT_SAVE_ARGS updated to use mode="overwrite" by default.
__init__:
- Validates mode/overwrite presence.
- Maps overwrite → mode internally.
save() dispatches based on mode:
- append → calls insert() if supported, else raises NotImplementedError.
- overwrite, error, ignore → handled via create_table/create_view with appropriate overwrite flag.
_describe() includes mode.

Credentials Support

Added credentials param to init signature and docstring.
Accepts either:
- A connection string (e.g. "duckdb:///my.db")
- A dict with connection params or a con string.
_connect() routes accordingly based on type.
_get_backend_name() extracts backend info for _describe().

All supported mode options were tested using both DuckDB and Postgres backends.
credentials parsing was tested for string, dict-with-con, and dict-with-backend formats.

Checklist

Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Updated jsonschema/kedro-catalog-X.XX.json if necessary
Added a description of this change in the relevant RELEASE.md file
Added tests to cover my changes
Received approvals from at least half of the TSC (required for adding a new, non-experimental dataset)

…ble materialization - feat: Introduces a `mode` parameter for save operations, allowing "append", "overwrite", "error", and "ignore" options to control write behavior. - Supports legacy `overwrite` flag with backward compatibility and deprecation warning. - Adds mode dispatching logic to handle different write scenarios. - Updates examples and docs to reflect new parameter. - Prevents simultaneous use of both `mode` and legacy `overwrite` to avoid ambiguity. - docs: Improves documentation and usage examples for new save modes. Semi-breaking change: Replaces `overwrite` save argument with `mode`; users should update configurations to use `mode`. Signed-off-by: gitgud5000 <[email protected]>

Signed-off-by: gitgud5000 <[email protected]>

deepyaman

Thanks for the PR!

Did a quick-pass review. I see you're still actively updating, so I'll revisit this later.

kedro-datasets/kedro_datasets/ibis/table_dataset.py

deepyaman · 2025-05-31T02:09:25Z

kedro-datasets/kedro_datasets/ibis/table_dataset.py


        self._materialized = self._save_args.pop("materialized")

+        # Handle mode / overwrite conflict


Wonder if insert + view needs to be disallowed; I haven't actually checked yet.

gitgud5000 · 2025-05-31T03:05:24Z

I've been activily working with this because I need the features now 😅.

I've also added support for passing credentials to the ibis.TableDataset to this PR I'll update the OP shortly to reflect this.
Is that ok or should I create a separate PR for this feature? @deepyaman

…precate connection - feat: introduce a new `credentials` parameter to accept connection info as a string or dict, preferred over the deprecated `connection` param - warn if both `credentials` and `connection` are provided, prioritizing `credentials` - support connection strings and dicts with connection strings for backend connections - feat: add backend name extraction to be used in `_describe()` method - docs: update docstrings to explain `credentials` and deprecation of `connection` !Semi- Breaking Change: deprecates the `connection` parameter in favor of `credentials` Signed-off-by: gitgud5000 <[email protected]>

…lacing the function dispatch dict with direct conditional handling, improving clarity and maintainability - docs: expands and clarifies docstring to explain available save modes and their effects, referencing Spark semantics for familiarity Signed-off-by: gitgud5000 <[email protected]>

Signed-off-by: gitgud5000 <[email protected]>

deepyaman · 2025-05-31T23:54:38Z

I've been activily working with this because I need the features now 😅.

Best reason. 😁

I've also added support for passing credentials to the ibis.TableDataset to this PR I'll update the OP shortly to reflect this. Is that ok or should I create a separate PR for this feature? @deepyaman

Sorry, I didn't check in time; ideal is a separate PR, but it's probably no big deal to have it in the same. If it ends up being a blocker to get something merged, can always split it out later.

- fix: add early return for empty DataFrame - fix: ensure table creation occurs before insert operations in append mode when table doesn't exist Signed-off-by: gitgud5000 <[email protected]>

gitgud5000 · 2025-08-18T12:55:24Z

Could you please take a look at this pull request, @ankatiyar @ravi-kumar-pilla?

ankatiyar · 2025-08-18T12:59:49Z

@gitgud5000 I'll take a look, in the meanwhile could you add some unit tests to go along with the changes? :D

- fix: remove lingering 'mode' key when mapping legacy overwrite to prevent unexpected writer kwargs - fix: treat empty pandas DataFrame as a no-op in save, supporting both pandas and ibis tables Prevents accidental parameter leakage and avoids errors when saving empty data. Signed-off-by: gitgud5000 <[email protected]>

…and legacy overwrite behavior Signed-off-by: gitgud5000 <[email protected]>

… new save modes and credentials parameter Signed-off-by: gitgud5000 <[email protected]>

…-support

Signed-off-by: gitgud5000 <[email protected]>

gitgud5000 · 2025-08-18T15:32:56Z

@ankatiyar Done ✅

I wasn’t able to resolve a couple of things:

All the tests I added pass locally, except for test_connection_config with the mssql config.

=========================== short test summary info ============================
FAILED kedro-datasets/tests/ibis/test_table_dataset.py::TestTableDataset::test_connection_config[None-None-None-connection_config1-key1]

The lint check pipeline is still failing, and I’m not sure why or how to fix it. Would appreciate your guidance 🙂

Signed-off-by: Ankita Katiyar <[email protected]>

Signed-off-by: gitgud5000 <[email protected]>

deepyaman · 2025-09-18T17:08:25Z

@gitgud5000 Sorry for the delay; I'll review in the next few days!

Signed-off-by: gitgud5000 <[email protected]>

Signed-off-by: Deepyaman Datta <[email protected]>

This reverts commit e4636b7. Signed-off-by: Deepyaman Datta <[email protected]>

Signed-off-by: Deepyaman Datta <[email protected]>

deepyaman · 2025-09-23T19:54:36Z

@gitgud5000 No rush at all (will focus on getting this PR merged first), but whenever you get a chance you can create one more PR off main with a git cherry-pick d922af12211e301dd53245bee0d08aa5c6e59852 (there are some minor conflicts, but they look easy to resolve). This will include all the changes unrelated to adding configurable save mode (credentials, empty datasets, etc.). If need to split that up more later can do it.

The main reason I'm asking you to create the PR instead of doing it myself is because then only one other person needs to review it other than myself. :)

deepyaman

Excited to have this long-awaited functionality!

deepyaman · 2025-09-23T23:13:23Z

kedro-datasets/kedro_datasets/ibis/table_dataset.py


+        # Handle mode/overwrite conflict.
+        if save_args and "mode" in save_args and "overwrite" in self._save_args:
+            raise ValueError("Cannot specify both 'mode' and deprecated 'overwrite'.")


There's actually no DeprecationWarning, but I'm also not 100% sure there needs to be one. However, if there isn't a DeprecationWarning, it doesn't make sense to call it "deprecated", and it shouldn't be on the list of breaking changes in the release notes.

deepyaman · 2025-09-23T23:15:25Z

kedro-datasets/RELEASE.md

+
+- `ibis.TableDataset`: Deprecated `save_args.overwrite` and the `connection` parameter in favor of `save_args.mode` and `credentials`. Using both `overwrite` and `mode` together raises an error; providing both `credentials` and `connection` emits a deprecation warning. The deprecated options will be removed in a future release.


Suggested change

- `ibis.TableDataset`: Deprecated `save_args.overwrite` and the `connection` parameter in favor of `save_args.mode` and `credentials`. Using both `overwrite` and `mode` together raises an error; providing both `credentials` and `connection` emits a deprecation warning. The deprecated options will be removed in a future release.

I don't think it's a breaking change even if raise a DeprecationWarning for using 'overwrite' (separate issue about actually raising the DeprecationWarning; see below).

(Also, the credentials and connection bit shouldn't have been left in here when I split out the PR; my bad.)

deepyaman · 2025-09-23T23:16:31Z

kedro-datasets/RELEASE.md

 - Fixed `PartitionedDataset` to reliably load newly created partitions, particularly with `ParallelRunner`, by ensuring `load()` always re-scans the filesystem .
 - Add a parameter `encoding` inside the dataset `SQLQueryDataset` to choose the encoding format of the query.
 - Corrected the `APIDataset` docstring to clarify that request parameters should be passed via `load_args`, not as top-level arguments.
+- Improved `_connect` and `_describe` for `ibis.TableDataset`; saving an empty pandas DataFrame is now a no-op.


Suggested change

- Improved `_connect` and `_describe` for `ibis.TableDataset`; saving an empty pandas DataFrame is now a no-op.

This was also supposed to get split out, sorry.

deepyaman · 2025-09-23T23:16:54Z

kedro-datasets/kedro_datasets/ibis/table_dataset.py

    from ibis import BaseBackend


+class SaveMode(StrEnum):


@gitgud5000 I took the liberty of refactoring save mode logic into an enum.

deepyaman · 2025-09-23T23:21:40Z

kedro-datasets/kedro_datasets/ibis/table_dataset.py

+    APPEND = auto()
+    OVERWRITE = auto()
+    ERROR = auto()
+    ERRORIFEXISTS = auto()


Do we need both "error" and "errorifexists"? "errorifexists" is a bit more explicit, but also a bit more verbose.

I decided to check if one or the other is preferred by Spark (maybe we could just use that), but it turns out it's not so simple. https://issues.apache.org/jira/browse/SPARK-21640 added "errorifexists" to the Python (and other API) options—"error" used to be the only option—but https://github.com/apache/spark/blob/v4.0.0/sql/api/src/main/java/org/apache/spark/sql/SaveMode.java only defines "errorifexists". Since we don't have the baggage, I'd be fine with just keeping "error" (feels simpler), but I also don't have any strong opinion against supporting both. Happy to have another maintainer (or @gitgud5000) weigh in.

deepyaman · 2025-09-23T23:23:05Z

kedro-datasets/pyproject.toml

 requires-python = ">=3.10"
 license = {text = "Apache Software License (Apache 2.0)"}
 dependencies = [
+    "backports.strenum; python_version < '3.11'",


It's a backport of a standard lib module; I don't personally think it's a big deal to add this dependency for everything, and just for Python 3.10.

deepyaman · 2025-09-23T23:23:46Z

kedro-datasets/tests/ibis/test_table_dataset.py

+def table_dataset(table_name, database_name, connection_config, load_args, save_args):
    return TableDataset(
-        table_name="test",
+        table_name=table_name,


@gitgud5000 Was making this parametrizable necessary? Just curious.

@gitgud5000 So, I ended up removing this. I see why it's necessary, but I think the correct thing to do is to clean up the environment by deleting any tables created (FWIW that's how Ibis tests also handle it). I've refactored it as such.

Signed-off-by: Deepyaman Datta <[email protected]>

gitgud5000 force-pushed the ibis-dataset-savemode-support branch from 2e9e539 to a41bb19 Compare May 31, 2025 01:41

chore: reformat code with Ruff for consistency

1f609c5

Signed-off-by: gitgud5000 <[email protected]>

gitgud5000 force-pushed the ibis-dataset-savemode-support branch from a41bb19 to 1f609c5 Compare May 31, 2025 01:44

deepyaman reviewed May 31, 2025

View reviewed changes

gitgud5000 changed the title ~~feat(datasets): ibis.TableDataset add configurable save mode for table materialization~~ feat(datasets): ibis.TableDataset add configurable save mode for table materialization and credentials support May 31, 2025

gitgud5000 added 3 commits May 31, 2025 03:51

fix: added missing 'errorifexists' to conditions in save modes

6445239

Signed-off-by: gitgud5000 <[email protected]>

gitgud5000 force-pushed the ibis-dataset-savemode-support branch from d545871 to 6445239 Compare May 31, 2025 03:52

Merge branch 'main' into ibis-dataset-savemode-support

256795d

ankatiyar requested a review from deepyaman June 3, 2025 14:52

gitgud5000 marked this pull request as draft June 4, 2025 11:13

gitgud5000 marked this pull request as ready for review August 15, 2025 18:49

fix(datasets): handle empty DataFrames and improve append mode

6130581

- fix: add early return for empty DataFrame - fix: ensure table creation occurs before insert operations in append mode when table doesn't exist Signed-off-by: gitgud5000 <[email protected]>

gitgud5000 force-pushed the ibis-dataset-savemode-support branch from 40bbfe3 to 6130581 Compare August 15, 2025 20:15

gitgud5000 added 2 commits August 18, 2025 10:31

test(tabledataset): add tests for save modes: append, error, ignore, …

55b8666

…and legacy overwrite behavior Signed-off-by: gitgud5000 <[email protected]>

gitgud5000 force-pushed the ibis-dataset-savemode-support branch from db200f3 to 55b8666 Compare August 18, 2025 14:31

gitgud5000 added 4 commits August 18, 2025 10:40

docs(RELEASE.md): update major features and improvements section with…

5a84f44

… new save modes and credentials parameter Signed-off-by: gitgud5000 <[email protected]>

Merge remote-tracking branch 'origin/main' into ibis-dataset-savemode…

7b04389

…-support

fix lint

6cfb8fe

Signed-off-by: gitgud5000 <[email protected]>

docs(table_dataset): fix indentation in docstring

2f50c6d

Signed-off-by: gitgud5000 <[email protected]>

Fix lint

4c6ddb9

Signed-off-by: Ankita Katiyar <[email protected]>

gitgud5000 force-pushed the ibis-dataset-savemode-support branch from 63b95c1 to 4c6ddb9 Compare August 18, 2025 16:38

gitgud5000 force-pushed the ibis-dataset-savemode-support branch from 0cbaabd to 940d901 Compare September 10, 2025 07:28

Fixes credential reference when constructing connection

3c836e3

Signed-off-by: gitgud5000 <[email protected]>

gitgud5000 requested a review from deepyaman September 10, 2025 07:53

gitgud5000 and others added 3 commits September 19, 2025 09:03

fix: remove unused pandas import from table_dataset.py

398d292

Signed-off-by: gitgud5000 <[email protected]>

chore(datasets): move notes under upcoming release

50acb2d

Signed-off-by: Deepyaman Datta <[email protected]>

revert(datasets): remove changes unrelated to mode

e4636b7

Signed-off-by: Deepyaman Datta <[email protected]>

deepyaman force-pushed the ibis-dataset-savemode-support branch from a3c387b to e4636b7 Compare September 23, 2025 19:35

deepyaman added 2 commits September 23, 2025 13:50

feat(datasets): add back changes unrelated to mode

d922af1

This reverts commit e4636b7. Signed-off-by: Deepyaman Datta <[email protected]>

revert(datasets): remove changes unrelated to mode

6a23b81

Signed-off-by: Deepyaman Datta <[email protected]>

deepyaman force-pushed the ibis-dataset-savemode-support branch from a245267 to 6a23b81 Compare September 23, 2025 19:50

deepyaman force-pushed the ibis-dataset-savemode-support branch from f29fe5c to f819670 Compare September 23, 2025 23:10

deepyaman approved these changes Sep 23, 2025

View reviewed changes

deepyaman force-pushed the ibis-dataset-savemode-support branch from f819670 to cda534a Compare September 23, 2025 23:30

refactor(datasets): define save mode using an enum

513aab0

Signed-off-by: Deepyaman Datta <[email protected]>

deepyaman force-pushed the ibis-dataset-savemode-support branch from 1057b14 to 513aab0 Compare September 23, 2025 23:44

deepyaman changed the title ~~feat(datasets): ibis.TableDataset add configurable save mode for table materialization and credentials support~~ feat(datasets): make table write mode configurable Sep 23, 2025

deepyaman added 9 commits September 23, 2025 22:08

chore(datasets): remove notes unrelated to this PR

2629069

Signed-off-by: Deepyaman Datta <[email protected]>

chore(datasets): reword release note on save modes

c4534e5

Signed-off-by: Deepyaman Datta <[email protected]>

refactor(datasets): simplify branching in save()

48c5e85

Signed-off-by: Deepyaman Datta <[email protected]>

revert(datasets): remove extra blank added to file

c75643d

Signed-off-by: Deepyaman Datta <[email protected]>

test(datasets): clean up any tables that we create

b4ba79a

Signed-off-by: Deepyaman Datta <[email protected]>

test(datasets): condense multi-parametrizes to one

4299844

Signed-off-by: Deepyaman Datta <[email protected]>

test(datasets): check raises a narrower error type

5f9fc08

Signed-off-by: Deepyaman Datta <[email protected]>

test(datasets): test legacy overwrite with fixture

4946541

Signed-off-by: Deepyaman Datta <[email protected]>

chore(datasets): use <= instead of .issubset()

55ba8a0

Signed-off-by: Deepyaman Datta <[email protected]>

deepyaman force-pushed the ibis-dataset-savemode-support branch from 62d976a to 55ba8a0 Compare September 24, 2025 19:29

deepyaman requested review from ravi-kumar-pilla and ankatiyar September 26, 2025 17:15


		self._materialized = self._save_args.pop("materialized")

		# Handle mode / overwrite conflict


		- `ibis.TableDataset`: Deprecated `save_args.overwrite` and the `connection` parameter in favor of `save_args.mode` and `credentials`. Using both `overwrite` and `mode` together raises an error; providing both `credentials` and `connection` emits a deprecation warning. The deprecated options will be removed in a future release.

feat(datasets): make table write mode configurable #1093

Are you sure you want to change the base?

feat(datasets): make table write mode configurable #1093

Conversation

gitgud5000 commented May 31, 2025 • edited by deepyaman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Development notes

Save Mode Handling

Credentials Support

Checklist

Uh oh!

deepyaman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gitgud5000 commented May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deepyaman commented May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gitgud5000 commented Aug 18, 2025

Uh oh!

ankatiyar commented Aug 18, 2025

Uh oh!

gitgud5000 commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deepyaman commented Sep 18, 2025

Uh oh!

deepyaman commented Sep 23, 2025

Uh oh!

deepyaman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gitgud5000 commented May 31, 2025 •

edited by deepyaman

Loading

gitgud5000 commented May 31, 2025 •

edited

Loading

deepyaman commented May 31, 2025 •

edited

Loading

gitgud5000 commented Aug 18, 2025 •

edited

Loading