Pass the typing conformance suite

[carljm]

...or explicitly document the cases where we intentionally choose to diverge from it? (In which case we should advocate for a change to / relaxation of the spec.)

[AlexanderWillner]

Based on How Well Do New Python Type Checkers Conform? we see that ty 0.0.1-alpha.19 passes 20 tests

[oconnor663]

Carl has pointed out some overly-strict asserts in the conformance suite that we might want to submit a PR upstream for. Copying from chat:

the relevant tests (which we still fail, due to the over-specification of assert_type) are at https://github.com/python/typing/blob/main/conformance/tests/generics_defaults.py#L30 and following

for us, an unspecialized mention of the value NoNonDefaults is just a singleton "class object" type, which is a subtype of type[NoNonDefaults[str, int]], but not equivalent to it

but these conformance tests (IMO) over-specify here

they should instead just test that if you type-annotate something as NoNonDefaults, you get a type of NoNonDefaults[str, int] for that name (which is true for us)

https://play.ty.dev/e871464d-4c6e-4d71-b032-b055f0192997

Obviously that still uses assert_type -- I think the real problem isn't using assert_type, it's that "the type of a generic class object as a value" is not well-specified in the Python type system, and the conformance suite shouldn't implicitly assert that it must be a default-specialized type[] type

[carljm]

In #1762 (comment) @AlexWaygood suggests a workaround we could implement in our assert_type handling that could be an alternative to changing these assert_type usages upstream. (The rationale is that this issue doesn't only impact the conformance suite, it also impacts other people using assert_type to test their stubs.)

[tsudol-plaid]

Now that ty is in beta (congrats!!!!) are there plans to add it to the conformance test suite?

[zanieb]

Yep, we're planning on working on it.

[yann-combarnous]

Pyrefly ran the conformance suite for ty, compared to Mypy and Pyright: https://pyrefly.org/blog/typing-conformance-comparison/

It would be nice to have this regularly updated in #1889 .

[sharkdp]

Pyrefly ran the conformance suite for ty, compared to Mypy and Pyright: pyrefly.org/blog/typing-conformance-comparison

Yes, we've recently uploaded our results: python/typing#2208. You can always see the latest version here. We also have an inofficial page here if you want to see which exact tests are failing.

It would be nice to have this regularly updated in #1889 .

That issue should generally be up to date. If you see anything that is missing or outdated, please let us know.

[AlexWaygood]

We now upload our conformance results upstream and have detailed commentary in those uploaded results that describe exactly why we fail any test that is marked as "partial". I believe we also have open issues in the ty issue tracker for nearly all unsupported or only-partially-supported typing-spec features.

In any event, this issue is too broad to still be useful as a tracking issue, so I'm closing it now that we upload our results upstream 🎉