LSP and Language Servers

The Language Server Protocol defines language capabilities such as completion, go-to-definition, and diagnostics as a protocol between the editor and the language service. It reduces the repetitive work of different editors adapting to each language, but it does not require the language service to adopt a specific compiler, thread model, or database.

A language server can reuse a compiler frontend or maintain a dedicated analysis engine tailored for interactive use. It deals with source code that is constantly changing and often incomplete, requiring it to balance response speed, result completeness, and version correctness. IDE language features can also be implemented internally within the editor; LSP is not the only source.

The Protocol Defines Capabilities Before Exchanging Requests and Notifications

LSP uses a JSON-RPC message model. Requests have an ID and correspond to a result or an error; notifications have no corresponding response. During the initialization phase, both parties exchange capabilities and then use features according to the negotiated results. For example, textDocument/publishDiagnostics is a notification sent by the server and should not be listed as a standard query initiated by the editor.

Document synchronization typically includes notifications such as didOpen, didChange, and didClose. Whether to use the full text or incremental edits is determined by the capability agreement. For open documents, the server must analyze the unsaved content synchronized by the editor, rather than re-reading the old file from disk every time. LSP 3.17 Specification

Understanding Analysis Dependencies via a Field Completion

When a user types account., the expression is incomplete. The analysis engine can retain the receiver account, the dot, and the missing member node, continue parsing the binding and type of account, and then enumerate candidate members. Candidates may also be constrained by visibility, generic substitution, traits, extension methods, and expected types.

Therefore, completion is not just "listing all names in the symbol table." The syntactic position determines which type of candidate is expected, name resolution determines the specific declaration, type analysis determines which members are available, and sorting and auto-imports improve the interactive experience.

Compilers can also perform error recovery rather than stopping at the first incomplete expression. The difference lies mainly in the goal: interactive systems more frequently require providing locally useful results even with erroneous source code.

Common Capabilities Quick Reference

Rename is not a global string replacement. Variables with the same name in different scopes are not the same symbol; names in macro expansions may not have directly modifiable source locations. After the server generates a set of edits, the client still needs to apply them according to the protocol and version state.

Incremental Computation: Caching Query Dependencies, Not Just Files

Consider the query chain: "file text → syntax tree → declaration signature → function body type → diagnostics." Modifying a local expression within a function body may invalidate that function's type result without affecting the exported signature; modifying a public type may affect multiple dependent files.

An incremental engine records dependencies and input versions, recalculating only affected queries, and can prevent invalidation from propagating further if the result hasn't changed. "Recalculating only the changed file" is too narrow, while "recalculating the entire project on any change" is too broad. Project configuration, compilation options, dependency versions, and generated files are also inputs.

The architecture of rust-analyzer separates syntax, semantics, and incremental queries, reflecting this organization for interactive analysis; this does not mean all results must be persisted to disk indexes. rust-analyzer Architecture

Project Indexing: Connecting Declarations and Usages via Symbol Identity

Project-level indexes help find symbols, references, and relationships in files that are not currently open. It can combine dynamic results from currently open files, background-built indexes, and optional external indexes. Caching or disk persistence are performance measures, not mandatory requirements of LSP.

clangd combines indexes from different sources, balancing the current editing state with information about the entire project. clangd Indexing Design

Indexes must interpret source code according to the actual compilation context. For C/C++, the same file may represent different programs under different macro definitions or include paths; using only file modification time as a cache key may reuse incorrect results. When indexing is not yet complete, it should be clear which queries can return partial results and which operations need to wait for more information.

Position Information: Characters Do Not Always Equal Bytes

LSP positions are typically represented by zero-based line numbers and line offsets. 3.17 supports negotiating position encoding; if no other encoding is negotiated, the UTF-16 compatible default is retained. The number of UTF-8 bytes, Unicode code points, and UTF-16 code units may differ.

For example, a non-BMP character occupies two code units in UTF-16 but may occupy four bytes in UTF-8. If the server directly treats byte offsets as LSP characters, completion replacements or diagnostic ranges near Chinese characters and emojis will be offset. The order of applying edits, newline representation, and range endpoints should also comply with the protocol. See the LSP Meta Model for specific data structures.

Sequential Edits Use the Intermediate Text

Suppose version 7 contains the single line a😀b, with UTF-16 position encoding. The a occupies one unit and the emoji two, so b starts at offset 3, although its Unicode code-point index is 2. Inserting X at offset 1 produces aX😀b, moving b to UTF-16 offset 4. Replacing range [4,5) with B then produces aX😀B.

Changes within one didChange notification are also applied in array order: the second range refers to the intermediate text, not the original version 7. The message's version denotes the state after all changes. See the document synchronization definition.

This conversion model handles one line of valid Unicode. Positions beyond line length clamp to the end. It rejects positions inside a surrogate pair to avoid splitting the emoji; that is this model's policy, not a claim that every implementation must repair such positions identically.

def index16(text, offset):
    if offset < 0:
        raise ValueError("negative position")
    units = 0
    for index, char in enumerate(text):
        if units == offset:
            return index
        units += len(char.encode("utf-16-le")) // 2
        if units > offset:
            raise ValueError("inside surrogate pair")
    return len(text)  # At or beyond the end of the line


def apply(text, changes):
    for start, end, replacement in changes:
        if start > end:
            raise ValueError("reversed range")
        lo, hi = index16(text, start), index16(text, end)
        text = text[:lo] + replacement + text[hi:]
    return text


assert index16("a😀b", 3) == 2
assert index16("a😀b", 99) == 3
assert apply("a😀b", [(1, 1, "X"), (4, 5, "B")]) == "aX😀B"
assert apply("a😀b", [(1, 3, "")]) == "ab"
for changes in [[(2, 3, "")], [(3, 1, "")], [(-1, 0, "")]]:
    try:
        apply("a😀b", changes)
    except ValueError:
        pass
    else:
        raise AssertionError("invalid edit accepted")

A complete implementation also needs line indexing, CRLF handling, multiline replacements and whole-document changes. After every edit, text and line indexes must agree; retaining old line starts while changing text corrupts subsequent conversions. See the Position definition.

Why does one cursor have three different offsets?

Three counters describe one character boundary, not three cursors. Crossing the emoji adds four UTF-8 bytes, two UTF-16 units and one code point. Inserting X moves b, so the next replacement needs the new UTF-16 range. Buttons select complete edit snapshots; the boundary slider observes without editing.

Semantic Highlighting vs. Syntax Highlighting

Regular expressions can identify local lexical patterns; Tree-sitter's highlights.scm primarily queries syntax tree nodes, rather than treating the entire file as a pure regex match. It can leverage syntactic structure but usually does not undertake complete cross-file type parsing. Tree-sitter Highlighting Documentation

LSP semantic tokens allow the server to return richer categories and modifiers, such as whether an identifier is a type, a parameter, or a read-only variable. It does not mandate "annotating every token in the source code," nor does it prohibit editors from retaining syntax highlighting as a fast base layer.

Scheduling: Fast Response and Correct Versions Are Equally Important

Interactive queries such as completion and hover should generally take precedence over full-project indexing. Background analysis can merge nearby edits, reuse snapshots, and cancel outdated tasks after input changes; however, it must not allow old diagnostics to overwrite new results when they arrive late.

$/cancelRequest expresses the cancellation of a specific protocol request, not a unified stop mechanism for arbitrary internal background tasks. Cancellation may race with completion; the server still needs to properly terminate the request; internal analysis also requires its own cooperative cancellation and resource management.

There are trade-offs between debounce waiting and query latency. Some fast local analyses can be executed on every edit; expensive background diagnostics are better suited for merging. There is no conclusion that "parsing on every keypress will definitely cause lag" or that "a unified 50ms debounce guarantees all requests stay below 100ms."

Why Cancellation Still Requires Result Validation

Request 81 analyzes v7; the document then becomes v8 and request 82 returns first. Even if cancellation of 81 has been requested, its computation may already have finished or its response may already be in flight. Cancellation does not promise that no response for that ID will arrive; the request still needs a terminating response. See request cancellation.

An internal task can retain (URI, document version, project configuration generation). The following conservative policy accepts a result only while its full snapshot key still matches. These fields are internal scheduling context, not a claim that LSP completion requests directly carry all of them. Configuration changes can invalidate a result even without a text edit.

current = ("file:///demo.lang", 8, 3)
pending = {81: ("file:///demo.lang", 7, 3), 82: current}
shown = []


def finish(request_id, result):
    snapshot = pending.pop(request_id)  # Complete bookkeeping for this request first
    if snapshot == current:
        shown.append(result)


finish(82, "new completion")
finish(81, "old completion")
assert shown == ["new completion"] and not pending
pending[83] = current
current = ("file:///demo.lang", 8, 4)  # Text unchanged; compilation configuration changed
finish(83, "stale configuration")
assert shown == ["new completion"]

This does not require every edit to discard every result. A more precise engine can prove that dependencies did not change or map old positions to a new version. A larger request ID alone proves neither: IDs associate requests with responses, rather than ordering document versions.

Finishing a reference search does not prove rename targets have stayed unchanged. Where supported, a versioned TextDocumentEdit records the basis for an edit and the client handles it according to versions and capabilities. Cross-file changes must also account for failure-handling capabilities; clients do not all provide identical atomic application semantics.

Does a larger request ID imply a fresher result?

The second task intentionally has a smaller request ID, preventing numeric order from masquerading as version order. Tasks retain launch-time text/config versions and validate against the current snapshot at delivery; stale results cannot replace accepted fresh results. Cancellation is intent; late delivery models an already-completed response race. Config changes also invalidate unchanged text results.

Learning Path for Minimal Implementation

First, implement initialization, document synchronization, and position conversion, ensuring the server holds the correct text; then provide local syntax diagnostics and simple hover; next, introduce name resolution, project context, and incremental queries; finally, implement cross-file references, renaming, and more complex completions. Each step should check error-prone source code and continuous edits, not just one-time complete files.

Related basics: AST and Error Recovery, Symbol Tables and Scopes.

References

• LSP 3.17 Specification.
• rust-analyzer Architecture.
• clangd Indexing.

Keywords: LSP, language server, JSON-RPC, incremental analysis, document version, UTF-16, semantic tokens, cancellation