Faster tokenizer#11392

Open

bschnurr wants to merge 18 commits intomicrosoft:mainfrom

bschnurr:faster-tokenizer

Commits on Apr 15, 2026

initial optimization
bschnurr
committed
Summary: The migration to the no-regex tokenizer had dropped support for : in type: ignore[...] bracket codes (e.g. ty:unresolved-reference). The fix adds Char.Colon to the allowed bracket-content …
bschnurr
committed
## Summary
bschnurr
committed

Commits on Apr 16, 2026

Fixed the regression in tokenizer.ts so matchIgnoreDirective now rejects [type: ignoretype: ignore[ or pyright: ignore[ when the closing ] is missing, instead of silently treating them as bare igno…
bschnurr
committed

Commits on Apr 17, 2026

Commits on Apr 18, 2026