[red-knot] Allow explicit specialization of generic classes #17023

dcreager · 2025-03-27T20:25:22Z

This PR lets you explicitly specialize a generic class using a subscript expression. It introduces three new Rust types for representing classes:

NonGenericClass
GenericClass (not specialized)
GenericAlias (specialized)

and two enum wrappers:

ClassType (a non-generic class or generic alias, represents a class type at runtime)
ClassLiteralType (a non-generic class or generic class, represents a class body in the AST)

We also add internal support for specializing callables, in particular function literals. (That is, the internal Type representation now attaches an optional specialization to a function literal.) This is used in this PR for the methods of a generic class, but should also give us most of what we need for specializing generic functions (which this PR does not yet tackle).

github-actions · 2025-03-27T20:32:12Z

`mypy_primer` results

Changes were detected when running on open source projects

async-utils (https://github.com/mikeshardmind/async-utils)
- error[lint:non-subscriptable] /tmp/mypy_primer/projects/async-utils/src/async_utils/scheduler.py:75:38: Cannot subscript object of type `Literal[_Task]` with no `__class_getitem__` method
- error[lint:invalid-argument-type] /tmp/mypy_primer/projects/async-utils/src/async_utils/scheduler.py:148:30: Object of type `T` cannot be assigned to parameter 3 (`payload`) of bound method `__init__`; expected type `T`
- error[lint:invalid-argument-type] /tmp/mypy_primer/projects/async-utils/_misc/_queue_testing.py:131:24: Object of type `None` cannot be assigned to parameter 2 (`item`) of bound method `sync_put`; expected type `T`
- error[lint:non-subscriptable] /tmp/mypy_primer/projects/async-utils/src/async_utils/_qs.py:510:16: Cannot subscript object of type `Literal[BaseQueue]` with no `__class_getitem__` method
- error[lint:non-subscriptable] /tmp/mypy_primer/projects/async-utils/src/async_utils/_qs.py:538:20: Cannot subscript object of type `Literal[BaseQueue]` with no `__class_getitem__` method
- error[lint:non-subscriptable] /tmp/mypy_primer/projects/async-utils/src/async_utils/_qs.py:566:24: Cannot subscript object of type `Literal[BaseQueue]` with no `__class_getitem__` method
- Found 38 diagnostics
+ Found 32 diagnostics

* main: [red-knot] Add property tests for callable types (#17006) [red-knot] Disjointness for callable types (#17094) [red-knot] Flatten `Type::Callable` into four `Type` variants (#17126) mdtest.py: do a full mdtest run immediately when the script is executed (#17128) [red-knot] Fix callable subtyping for standard parameters (#17125) [red-knot] Fix more `redundant-cast` false positives (#17119) Sync vendored typeshed stubs (#17106) [red-knot] support Any as a class in typeshed (#17107) Visit `Identifier` node as part of the `SourceOrderVisitor` (#17110) [red-knot] Don't infer Todo for quite so many tuple type expressions (#17116) CI: Run pre-commit on depot machine (#17120) Error instead of `panic!` when running Ruff from a deleted directory (#16903) (#17054) Control flow graph: setup (#17064) [red-knot] Playground improvements (#17109) [red-knot] IDE crate (#17045) Update dependency vite to v6.2.4 (#17104) [red-knot] Add redundant-cast error (#17100) [red-knot] Narrowing on `in tuple[...]` and `in str` (#17059)

Co-authored-by: Alex Waygood <[email protected]>

dcreager

Other than the tracked structs Salsa bug, this is ready for review!

dcreager · 2025-04-07T22:32:46Z

crates/red_knot_python_semantic/resources/mdtest/generics/classes.md

+reveal_type(Bounded[int]())  # revealed: Bounded[int]
+reveal_type(Bounded[IntSubclass]())  # revealed: Bounded[IntSubclass]
+
+# error: [invalid-argument-type] "Object of type `str` cannot be assigned to parameter 1 (`T`) of class `Bounded`; expected type `int`"


The diagnostics look like this because I'm reusing the call binding mechanism to match parameters and check types. It would be nice for this diagnostic to talk about type parameters, but would prefer to do that as a follow-on.

dcreager · 2025-04-07T22:36:02Z

crates/red_knot_python_semantic/resources/mdtest/properties.md

+> TODO: This test is currently disabled pending
+> [an upstream Salsa fix](https://github.com/salsa-rs/salsa/pull/741). Once that has been merged,
+> re-enable this test by changing the language codes below back to `py`.


This depends on salsa-rs/salsa#741

If we want to unblock this PR on the salsa fix, the other thing we can do is temporarily revert the change that makes mdtests reuse the same Salsa db, and live with slow mdtests until the Salsa fix lands.

dcreager · 2025-04-07T22:45:58Z

crates/red_knot_python_semantic/src/types/class.rs

+    /// The member resolves to a member on the class itself or any of its proper superclasses.
+    ///
+    /// TODO: Should this be made private...?
+    pub(super) fn class_member(self, db: &'db dyn Db, name: &str) -> SymbolAndQualifiers<'db> {


Most of these methods exist on both ClassType and ClassLiteralType. The "real" implementations are on ClassLiteralType, since that's what represents a class body in the AST. The ClassType methods wrap that, applying the specialization in the case of a generic alias.

crates/red_knot_python_semantic/resources/mdtest/generics/classes.md

crates/red_knot_python_semantic/resources/mdtest/generics/scoping.md

sharkdp · 2025-04-08T13:36:28Z

and two enum wrappers:

* `ClassType` (a non-generic class or generic alias, represents a class _type_ at runtime)

Minor comment: is this outdated or am I misunderstanding something? Are you talking about Type enum variants here? If so, should it be Type::GenericAlias?

If this interpretation is correct, that would mean that non-generic class types also fall under Type::GenericAlias (which might be fine, I'm just trying to understand).

dcreager · 2025-04-08T13:59:01Z

Minor comment: is this outdated or am I misunderstanding something? Are you talking about Type enum variants here? If so, should it be Type::GenericAlias?

If this interpretation is correct, that would mean that non-generic class types also fall under Type::GenericAlias (which might be fine, I'm just trying to understand).

No, it (and ClassLiteralType) are new enums defined in types/class.rs:

ruff/crates/red_knot_python_semantic/src/types/class.rs

Lines 170 to 173 in 43554e2

    
           pub enum ClassType<'db> { 
        
               NonGeneric(NonGenericClass<'db>), 
        
               Generic(GenericAlias<'db>), 
        
           }

Type::ClassLiteral wraps ClassLiteralType. ClassType doesn't have a Type variant itself; instead it's used in e.g. Type::Instance and Type::SubclassOf, since an object is an instance of a non-generic class or of a specialized generic alias.

So a non-generic class (i.e. the class itself, not an instance of it) would be a

Type::ClassLiteral(ClassLiteralType::NonGeneric(NonGenericClass { .. }))

A generic class that hasn't been specialized would be a

Type::ClassLiteral(ClassLiteralType::Generic(GenericClass { .. }))

and a generic class that has been specialized would be a

Type::GenericAlias(GenericAlias { .. })

carljm

This is fantastic!

Haven't reviewed nearly all of it yet, but I'm going to switch focus to the Salsa bug, so submitting the comments I have.

crates/red_knot_python_semantic/resources/mdtest/generics/scoping.md

crates/red_knot_python_semantic/src/types/display.rs

carljm · 2025-04-08T14:48:19Z

crates/red_knot_python_semantic/resources/mdtest/properties.md

+> TODO: This test is currently disabled pending
+> [an upstream Salsa fix](https://github.com/salsa-rs/salsa/pull/741). Once that has been merged,
+> re-enable this test by changing the language codes below back to `py`.


If we want to unblock this PR on the salsa fix, the other thing we can do is temporarily revert the change that makes mdtests reuse the same Salsa db, and live with slow mdtests until the Salsa fix lands.

crates/red_knot_python_semantic/resources/mdtest/stubs/class.md

carljm

This is great!!

carljm · 2025-04-08T21:37:31Z

crates/red_knot_python_semantic/src/types/generics.rs

+        }
+    }
+
+    pub(crate) fn signature(self, db: &'db dyn Db) -> Signature<'db> {


What is your longer-term thinking about the reuse of Signature and call-binding for specialization? It seems to me that it maybe doesn't buy us very much, since type params are always positional-only, meaning we don't need most of the interesting code from call binding, and it will be hard to integrate e.g. correct handling of constrained typevars, and better diagnostics. (I guess what I'm saying here is I can see why this was a quick way to get something working, but I'm not convinced this is a path we want to go down, vs just writing separate custom code for specialization.)

That's a good point, though my original thinking was to make it lean on the existing call binding mechanism more:

ruff/crates/red_knot_python_semantic/src/types/infer.rs

Lines 5692 to 5695 in 43554e2

// TODO: Move this logic into a custom callable, and update `find_name_in_mro` to return

// this callable as the `__class_getitem__` method on `type`. That probably requires

// updating all of the subscript logic below to use custom callables for all of the _other_

// special cases, too.

i.e. explicit specialization would happen in a custom callable, which __getitem__ would resolve to on generic classes. Though that would require adding hooks to Signature to allow customized diagnostics.

crates/red_knot_python_semantic/src/types/generics.rs

crates/red_knot_python_semantic/src/types.rs

crates/red_knot_python_semantic/src/types/narrow.rs

crates/red_knot_python_semantic/src/types/property_tests/type_generation.rs

crates/red_knot_python_semantic/src/types/infer.rs

* origin/main: [red-knot] Default `python-platform` to current platform (#17183) [red-knot] Add new 'unreachable code' test case (#17306) [red-knot] mypy_primer: Run on `async-utils` (#17303) [red-knot] Add custom `__setattr__` support (#16748) [red-knot] Add `__init__` arguments check when doing `try_call` on a class literal (#16512) [`flake8-pie`] Avoid false positive for multiple assignment with `auto()` (`PIE796`) (#17274) [syntax-errors] Async comprehension in sync comprehension (#17177) [`airflow`] Expand module path check to individual symbols (`AIR302`) (#17278) [syntax-errors] Check annotations in annotated assignments (#17283) [syntax-errors] Extend annotation checks to `await` (#17282) [red-knot] Add support for `assert_never` (#17287) [`flake8-pytest-style`] Avoid false positive for legacy form of `pytest.raises` (`PT011`) (#17231) [red-knot] Do not show types for literal expressions on hover (#17290) [red-knot] Fix dead-code clippy warning (#17291) [red-knot] Reachability analysis (#17199) [red-knot] Don't use latency-sensitive for handlers (#17227)

* main: [red-knot] Allow explicit specialization of generic classes (#17023) [`airflow`] Refactor `AIR301` logic and fix typos (`AIR301`) (#17293) [`airflow`] Extract `AIR312` from `AIR302` rules (`AIR302`, `AIR312`) (#17152) [red-knot] Improve handling of visibility constraints in external modules when resolving `*` imports (#17286) [red-knot] Add more tests for `*` imports (#17315)

…h#17023) This PR lets you explicitly specialize a generic class using a subscript expression. It introduces three new Rust types for representing classes: - `NonGenericClass` - `GenericClass` (not specialized) - `GenericAlias` (specialized) and two enum wrappers: - `ClassType` (a non-generic class or generic alias, represents a class _type_ at runtime) - `ClassLiteralType` (a non-generic class or generic class, represents a class body in the AST) We also add internal support for specializing callables, in particular function literals. (That is, the internal `Type` representation now attaches an optional specialization to a function literal.) This is used in this PR for the methods of a generic class, but should also give us most of what we need for specializing generic _functions_ (which this PR does not yet tackle). --------- Co-authored-by: Alex Waygood <[email protected]> Co-authored-by: Carl Meyer <[email protected]>

dcreager added the ty Multi-file analysis & type inference label Mar 27, 2025

dcreager added 6 commits March 30, 2025 12:00

Build context for generic classes

3be8257

Handle explicit specialization before outputting lints

4c76cb5

Explicitly specialize classes

8048604

Track those structs

fe76d56

Add Specialization

4346e7d

Add CallableType::Specialized

2380ade

dcreager force-pushed the dcreager/special-class branch 5 times, most recently from d0a457c to cd5b3fb Compare March 31, 2025 15:56

dcreager and others added 17 commits March 31, 2025 17:04

Use union to hold typevar constraints

178ee89

Add Type::TypeVar variant

feb1ea9

Fix failing tests

3c1ad79

doc Type::TypeVar

59430b6

More correct handling of final bounds/constraints

77ffa80

use list[T] so generic funcs are callable even with Never

6f86720

lint

cf81967

Add (currently failing) narrowing tests

6615df1

Typevars _can_ be fully static I guess

b2f5a2a

Simplify intersections with constrained typevars

590680c

Fix tests

debd60a

lint

aa391fd

Update crates/red_knot_python_semantic/src/types/type_ordering.rs

e57e62e

Co-authored-by: Alex Waygood <[email protected]>

Clarify that typevar is subtype of object too

3df79cc

Clarify non-fully-static bounded typevars aren't subtypes

5b08e93

Add more tests for constrained gradual typevars

82e810f

Don't specialize function twice

7aaeb47

dcreager commented Apr 7, 2025

View reviewed changes

fix docs

43554e2

dcreager marked this pull request as ready for review April 7, 2025 22:52

dcreager requested review from carljm, AlexWaygood and sharkdp as code owners April 7, 2025 22:52

sharkdp reviewed Apr 8, 2025

View reviewed changes

crates/red_knot_python_semantic/resources/mdtest/generics/classes.md Outdated Show resolved Hide resolved

sharkdp reviewed Apr 8, 2025

View reviewed changes

crates/red_knot_python_semantic/resources/mdtest/generics/scoping.md Outdated Show resolved Hide resolved

carljm reviewed Apr 8, 2025

View reviewed changes

AlexWaygood mentioned this pull request Apr 8, 2025

[red-knot] Improve handling of visibility constraints in external modules when resolving * imports #17286

Merged

dcreager mentioned this pull request Apr 8, 2025

[red-knot] Super-basic generic inference at call sites #17301

Merged

carljm approved these changes Apr 9, 2025

View reviewed changes

dcreager added 11 commits April 9, 2025 09:46

Better TODO fallback type

7c3405a

Generic aliases are literals in type display

6593d90

Explain self_instance not being specialized

7ca6a60

Comment other non-specializations

dea493e

Add xfail for generic method inside generic class

048bb8b

More Python-like displays for specializations

06859fa

Narrow type(generic) better

d138405

Better todos

0997eca

Add TODO about property test data

6ab1b90

lint

bcb147f

dcreager merged commit ff376fc into main Apr 9, 2025
23 checks passed

dcreager deleted the dcreager/special-class branch April 9, 2025 15:18

	// TODO: Move this logic into a custom callable, and update `find_name_in_mro` to return
	// this callable as the `__class_getitem__` method on `type`. That probably requires
	// updating all of the subscript logic below to use custom callables for all of the _other_
	// special cases, too.

[red-knot] Allow explicit specialization of generic classes #17023

[red-knot] Allow explicit specialization of generic classes #17023

Uh oh!

Conversation

dcreager commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mypy_primer results

Uh oh!

dcreager left a comment

Choose a reason for hiding this comment

Uh oh!

dcreager Apr 7, 2025

Choose a reason for hiding this comment

Uh oh!

dcreager Apr 7, 2025

Choose a reason for hiding this comment

Uh oh!

carljm Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

dcreager Apr 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sharkdp commented Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dcreager commented Apr 8, 2025

Uh oh!

carljm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

carljm Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

carljm left a comment

Choose a reason for hiding this comment

Uh oh!

carljm Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

dcreager Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dcreager commented Mar 27, 2025 •

edited

Loading

github-actions bot commented Mar 27, 2025 •

edited

Loading

`mypy_primer` results

sharkdp commented Apr 8, 2025 •

edited

Loading