api: plan for narwhals.stable.v2

[MarcoGorelli]

I think we can aim for narwhals.stable.v2 some time in 2025

The main things I'd like to achieve are:

• remove "interchange" level of support, and instead only have:
  • "full": must support Series and have a defined row order (pandas, polars.DataFrame, pyarrow.Table, modin.DataFrame, cudf.DataFrame)
  • "lazy": must support collect, adding/removing/subsetting columns, filtering rows. Initially, at least, row-order-dependent operations (such as nw.cum_sum) will be left out. This will include Polars.LazyFrame, DuckDBPyRelation, pyspark.DataFrame, ibis.Table, dask.DataFrame. Later we can add an order_by argument to these functions which will become required when working with nw.Lazyframe
• make the breaking changes listed in breaking changes

Candidate changes:

• rework from_native. It will get simpler once we remove interchange-level things, but it's still fairly complicated. Not sure how much I like it to be completely honest, but I also haven't come up with anything better

Usage of narwhals.stable.v1 should remain unaffected

Tentative date: some time in 2025

---

Why remove support for the dataframe interchange protocol?

• the project has received no updates in over a year, issues and requests go unanswered, and it has no governance model in place so nobody is taking responsibility for it
• only a handful of dataframe libraries implement the protocol, and for most of them, we already provide (or are working to provide) native support in Narwhals. The only exception is Vaex, which itself seems to be deprecated (no releases in ~1.5 years), but we could support that too if there was demand
• from a Narwhals usage perspective, it adds a lot of complexity to have 3 layers of support. It's much simpler to just have: full, and lazy.

---

Outstanding items

Breaking changes:

• deprecate order-dependent expressions being passed to nw.LazyFrame
• deprecate length-changing order-dependent expressions (Expr.sort, Expr.gather_every, Expr.head, Expr.tail, Expr.sample)
• introduce restrictions on when length-changing expressions can be used in lazy API
• deprecate eager_or_interchange_only from from_native
• all expressions passed to group_by(...).agg should aggregate
• sort out [Bug]: Pyspark reduction does not preserve names root names with nw.all() #1780 (to make sure that addressing it doesn't require further API breaks)
• does nw.LazyFrame really need .clone? feat!: require at least one expression be passed to lazyframe select and lazyframe.with_columns, remove lazyframe.clone #2206
• native_namespace -> backend in IO functions Deprecate native_namespace in favour of backend in IO functions #1888
• deprecateLazyFrame.gather_every
• should pd.Categorical get mapped to nw.Enum? feat: let nw.Enum accept categories, map pandas ordered categorical to Enum (only in main namespace, not stable.v1) #2192
• time zones in duckdb api: check whether we should get the connection time zone for duckdb/pyspark #2165
• empty sum: 0 or null?
• expression expansion in group-by api: formalise expression expansion in group-by #2225
• LazyFrame.with_row_index: [Enh]: Allow LazyFrame.with_row_index(..., order_by=...) #2307
• horizontal concatenation disallow concat(how='horizontal') for lazyframes #2340
• IO functions API: io functions for v2 #2116
• utils public/private: api: move non-public parts of utils.py to private module(s) #2634
• stricter is_*_dataframe feat: raise informative error or warning when passing narwhals object to nw.dependencies.is_*_dataframe #1437
• deprecate get_level from main namespace
• any/all horizontal ignore_nulls API: align any_horizontal and all_horizontal with Polars' future behaviour? #2678
• LazyFrame.schema should LazyFrame.schema issue a performance warning? #2783

not considered blockers, as they don't change backcompat:

• IO functions API: io functions for v2 #2116
• keep argument in mode feat: add optional keep argument to Expr.mode #1793

[MarcoGorelli]

Thanks for looking into this! As far as I understood, this would not be user-facing, right? If so, definitely on-board with refactoring

[dangotbanned]

Thanks for looking into this! As far as I understood, this would not be user-facing, right? If so, definitely on-board with refactoring

Great @MarcoGorelli

Yeah I don't think any of this would be user-facing.

Making changes like this for v2 would impact internals and probably extension authors.
To what extent though depends on what things you might want to move around

[MarcoGorelli]

that's fine then, the extensions mechanism is still marked as experimental (and I don't know of anyone who's actually tried extending Narwhals yet)

[dangotbanned]

Achieves the same goal, but you get a context that is evaluated once per-class (e.g. enforce some error handling at declaration-time

One attribute that wouldn't work with this is backend_version: tuple[int, ...].
AFAICT, for that you'd want something more like a write-once attribute, per-implementation

narwhals/narwhals/utils.py

Lines 387 to 398 in f79a9ea

	MIN_VERSIONS: dict[Implementation, tuple[int, ...]] = {
	Implementation.PANDAS: (0, 25, 3),
	Implementation.MODIN: (0, 25, 3),
	Implementation.CUDF: (24, 10),
	Implementation.PYARROW: (11,),
	Implementation.PYSPARK: (3, 5),
	Implementation.POLARS: (0, 20, 3),
	Implementation.DASK: (2024, 8),
	Implementation.DUCKDB: (1,),
	Implementation.IBIS: (6,),
	Implementation.SQLFRAME: (3, 14, 2),
	}

You can't know what version is installed (if any) at class-definition time.
However, you would know that the version cannot change after it has been seen once.

So for something like (#2007 (comment)), you can be sure backend_version is known after the first ArrowSeries is instantiated.

PandasLike... would probably need the most thought on if it could support this

[dangotbanned]

Using __init_subclass__ for Version in #2324 (comment)