Semantic tokens re-check files on a cold query checker after diagnostics already checked them #65386

Acknowledgement

• I acknowledge that issues using this template may be closed without further explanation at the maintainer's discretion.

Comment

Summary: after every edit, the language server type-checks the edited file twice. Diagnostics check it on the dedicated diagnostics checker. Semantic tokens then re-infer most of the same types on a fresh query checker, because reclassifyByType calls GetTypeAtLocation for every variable, property and parameter (ls/semantictokens.go:421). On a real component file that's ~40 ms of extra CPU per semantic tokens request. I'd like to know whether reusing the diagnostics checker, once it has finished with the file, fits the checker pool design.

Measurements (main @ f797316, Apple M1 Pro, recharts src/cartesian/Line.tsx: 1,093 lines in a 658-file program; 120 keystrokes, 3 runs)

About 28 ms of each semantic tokens request is first-touch type inference: destructured props, const initializers that are calls, large annotation types. It's all local to the file, so I couldn't find a per-token shortcut that keeps the classification identical.

Why reuse would usually apply: VS Code pulls diagnostics as soon as a document changes (diagnosticPullOptions.onChange: true in packages/vscode-typescript/src/client.ts). It requests semantic tokens at least 300 ms after a content change (REQUEST_MIN_DELAY = 300 in vscode's documentSemanticTokens.ts). So by the time semantic tokens arrive, the diagnostics checker has normally finished with the file.

What I tried (branch: hood/TypeScript perf/semantic-tokens, +116/−5)

The pool lends the diagnostics checker to a semantic tokens request only when all of these hold:

• the checker is idle (non-blocking diagSem);
• it has fully checked the file (type check and unused check, not cancelled);
• no diagnostics request for that file is still in flight.

The in-flight guard is needed because a diagnostics request uses the checker three times (semantic, suggestion and declaration diagnostics) and releases it in between. In every other case the request gets a query checker exactly as today, and it never waits. So queries never run on the diagnostics checker before or during a file's diagnostics, and the order in which the checker walks the file is unchanged. That ordering is what prevents bugs like #51661, where a query asked for a type before the normal walk and the result was an order-dependent false implicit-any in the editor.

Correctness so far:

• Semantic tokens and diagnostics for Line.tsx and three other open files are byte-identical to main across 600 responses, both sequential and concurrent. That includes TS2883 errors that print types, unused-variable suggestions, and the transient errors you get mid-typing.
• hereby test, lint and check:format pass.
• The new pool test fails if the in-flight guard is removed.

The part I'm not sure about: after the file is checked, the classification queries still create a few types and symbols on the diagnostics checker (~6 types and 32 symbols per request here). Most likely these are throwaway re-checks of expressions. I saw no change in any diagnostics, but I can't prove that holds for every program. If keeping that checker completely free of query work is a hard requirement, this approach doesn't fit, and I'd be glad to hear whether another direction would be welcome.

I worked on this with help from an AI coding assistant (Claude Code). I've reviewed the change, and I'll handle any follow-up myself.