11 min read #compilers #compilation-technology-applications
On this page

LSP and Language Servers

IDE completion, jump-to-definition, hover types, refactoring—these code intelligence features all come from the language server, which is essentially a compiler frontend + incremental updates + error-tolerant parsing. LSP exposes lexical/symbol table capabilities to the editor via JSON-RPC.

Overview

IDE code intelligence—completion, jump-to-definition, hover type display, refactoring—traditionally required each IDE to write a separate analysis logic for each language it supported (M × N). LSP (Language Server Protocol) solves this using the same strategy as MCP: provide one language server process per language, with the IDE communicating via JSON-RPC. For compilation technology, LSP is a direct consumer of compiler frontend data (lexical/syntax/semantic/symbol table)—a language server is essentially a compiler frontend + an incremental update engine. This article covers the core request types of the LSP protocol, how language servers use the compiler frontend techniques from previous chapters to build indexes and respond, and why "being fast enough" is harder to achieve than "being precise enough."

LSP is not a compiler; it's a compiler frontend with a different goal

What a compiler wants: produce machine code from source code; correctness is paramount; batch processing across a set of files. What a language server wants: update completion/diagnostics/highlighting in <100ms for every keystroke, provide reasonable results for incomplete/erroneous code, and only recompute changed parts.

The latter requirement means a language server cannot simply run the entire compiler pipeline—it must be incremental and tolerant of syntax errors (source code will inevitably have syntax errors while the user is still typing).

Core Request Types: All Queries on Symbol Tables + AST

Every LSP request is a query on compiler frontend data structures:

RequestCorresponding Compiler FrontendWhat is Queried
textDocument/completionSymbol Table + Type SystemWhat names are visible in the current scope? What are their types and documentation?
textDocument/definitionSymbol TableOn which line is the definition of this identifier?
textDocument/referencesSymbol Table (Reverse Lookup)Where is this definition referenced?
textDocument/hoverSymbol Table + Type SystemWhat is the type of this identifier? What does its documentation comment say?
textDocument/signatureHelpType SystemWhat are the parameter lists and overloads of the current function? Which parameter is currently active?
textDocument/renameSymbol Table (All References)Change the name at all reference sites of this definition to the new name.
textDocument/publishDiagnosticsSemantic Analysis (Type Checking)What type errors/unused variables exist in this file?

All capabilities stem from the Symbol Table. definition is a lookup in the symbol table, references is the use list of a symbol, completion is an enumeration of the symbol table in the current scope, and hover is the type and doc string of a symbol. If the symbol table does not maintain a reverse lookup index for "where this name is referenced," references and rename cannot be efficient—building the symbol table requires bidirectional indexes (from definition to reference, and from reference to definition).

Indexing: Knowing What Each File Exports Before Opening It

In large projects with thousands of files, when a user opens main.rs, the language server cannot parse the entire project. It needs a persistent index:

Persistent Index: Startup Load, Incremental Update, On-Demand Query Startup ① Load Existing Index From previously stored results Deserialize and load ② Incremental Update Re-parse files that changed since last close ③ Query Response Open new file: query external symbols Only parse that file in memory What the Index Stores Exported Symbols Symbols exported (pub) by each file: name, location, type signature, documentation Import List Import (use/import) list for each file, to quickly check "who references this symbol" Dependency Graph Symbol dependency graph: A calls B → file containing A depends on definition file of B The index allows the language server to avoid parsing the entire project when opening a file—only incrementally update changed files, while unaffected files are resolved via direct table lookup. This is key to <100ms response times.

rust-analyzer uses Salsa (an incremental computation framework) to maintain this index. Any change only recomputes affected files, while unaffected files are retrieved directly from the cache. This is the key to keeping IDE response times under <100ms—not full recomputation, but incremental updates.

Error-Tolerant Parsing: Producing Results on Syntactically Invalid Input

When a user is typing (let x = some_struct.), the source code is inevitably incomplete—the parser sees the dot after some_struct, expects a field name, but encounters EOF. A compiler would stop here with an error. A language server cannot stop; it must recover from the error and provide completion for what might follow the ..

This requires the error recovery mechanism described in AST Design and Error Recovery to be robust: upon seeing an incomplete expression, the parser should still construct an AST node for it (marked as an error). Higher-level semantic analysis can then perform partial type inference on this "AST with errors"—extracting the field list from the type information of some_struct and returning it as completion candidates.

User Input: let x = some_struct.
Parser Output: Expr::Field { object: "some_struct", field: <Error> }
Semantic Analysis: Look up type of some_struct in symbol table → StructFoo { a: i32, b: String }
Completion: [a: i32, b: String]

Without error recovery, there is no completion—these two are a pair in a language server. The only distinction is whether "the current token is a complete expression" or "the current token is a syntax error." The latter requires the AST node produced by the parser to mark "the field is missing," allowing semantic analysis to know that the user expects completion at that position.

Semantic Tokens: Another Way to Implement Syntax Highlighting

Traditional highlighting uses regex matching (like Tree-sitter's highlights.scm), but regex doesn't understand semantics—foo is a variable definition in let foo = ..., a function call in foo(), and a function declaration in fn foo(). LSP's textDocument/semanticTokens/full assigns a semantic category (variable, function, keyword, type, comment...) to each token, and the IDE colors them based on these categories.

The language server determines the semantic category for each token on top of the AST + symbol table—this requires the tokens produced by the lexer to carry sufficient information (is it an identifier? Does reverse lookup in the symbol table reveal it as a variable, function, or type?), or for semantic analysis to annotate them additionally.

Architecture: Single Process, Multi-threading, Incremental Scheduling

The typical architecture of industrial language servers (rust-analyzer, clangd):

Language Server Architecture: Single Process, Multi-threading, Incremental Scheduling Main Thread (IO) Receive JSON-RPC requests → Send to scheduler Scheduler Request queue, coalesce requests for the same file (debounce) → Assign to worker threads Worker Threads Parse file → Update index → Produce response Debounce User types 10 characters within 500ms, do not re-parse character by character—wait 50ms of no input before parsing Cancellation Previous parse not finished, new input arrived, discard previous work using $/cancelRequest Debounce merges "continuous input" into "parse only after the last input"; cancellation discards stale intermediate results— both are key to stable <100ms response times. It's not full recomputation, but doing only the necessary one.

Debounce and cancellation are lifelines for performance—when a user types quickly, on_type frequency is ~100ms per character. If every character triggered a full file re-parse, the language server would 100% freeze. Debounce merges "N consecutive inputs" into "parse after the last one," and cancellation discards outdated intermediate results.

References

  • LSP 3.17 specification: https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/
  • rust-analyzer: https://github.com/rust-lang/rust-analyzer — Architecture documentation and source code for the Rust LSP server
  • Clangd: https://clangd.llvm.org — C/C++ LSP server based on the Clang frontend

Keywords: LSP, Language Server Protocol, JSON-RPC, completion, go-to-definition, find-references, hover, diagnostics, semantic tokens, incremental parsing, index, debounce, cancellation, error-tolerant parsing, symbol table reverse lookup, rust-analyzer, clangd

Related pages

© 2026 liz's doc