Context Engineering

Models are stateless—everything they "know" resides in the context window of each request. Treat the window as a budget: cache stable prefixes, compress old history, and load tools on demand, rather than stuffing it indiscriminately. Window management is the primary variable for agent performance.

Overview

Large Language Models (LLMs) are stateless within a single request: they do not remember the previous conversation, nor do they have hidden "memory." Their entire understanding of the world is limited to the text provided in that specific request—the context window. "Context engineering" is the practice of deciding what to put in this limited, expensive, and position-sensitive window, where to place it, and how to reuse it, in order to achieve more accurate, faster, and cheaper outputs.

This is important because our intuition about LLMs is often misaligned: we assume that "feeding in more data" will always yield better answers. In reality, windows have an upper limit (Claude Opus 4.8 supports 1M tokens), attention computation scales approximately as ~O(n²) with length, and models often struggle to "see clearly" information in the middle of the window. Treating the window as a budget to be managed, rather than an bottomless trash can, is the first principle of using any modern model effectively.

Historically, this evolved from "prompt engineering" (how to phrase things): in the era of short, single-turn prompts, phrasing was everything; in the era of long contexts, tool calling, and multi-turn agents, what is in the window, the order in which it is arranged, and what can be cached are the dominant factors. Anthropic progressively introduced prompt caching, compaction, context editing, and adaptive thinking as API primitives between 2024 and 2026; these are essentially tools for context engineering.

The Model Only "Knows" the Context Window in One Round

For every /v1/messages request, the model sees a concatenated text block (rendered in the fixed order: tools → system → messages):

Key takeaway: The model "remembers" the previous turn only because the caller sent the entire history again. The API is stateless; "memory" is maintained by the caller. This also explains why long conversations become increasingly expensive—every turn requires resending the entire history.

Tokens and Attention: Why Longer is More Expensive and Slower

Text is first tokenized into tokens by the tokenizer (Claude's tokens ≠ OpenAI's tokens; do not use tiktoken to estimate Claude, as it will underestimate by 15–20%; use count_tokens for accuracy). Billing, rate limiting, and window limits are all measured in tokens.

Length brings two types of costs:

• Compute ~O(n²): Self-attention allows each token to attend to all other tokens; doubling the sequence length quadruples the computation. This is the root cause of "longer is slower."
• KV Cache VRAM: To avoid recomputing history for every generated token, inference engines cache the Key/Value tensors for each layer of the history. VRAM usage grows linearly with context length. This is especially tight for local deployments—see Local LLM Deployment: a 64k context f16 KV cache requires ~5.37 GB, which won't fit on a 24GB card, requiring quantization to Q8_0 to reduce it to ~2.85 GB. Cloud APIs don't expose this VRAM, but it is the physical reason why "long context is more expensive."

KV cache (tensors cached by the inference engine for acceleration) and prompt caching (prefix caching by the API to save money) discussed below are two different layers of "caching." Do not confuse them. The former accelerates within a single generation; the latter reuses prefixes across requests.

Context Budget: Who is Competing for this Window

Treat the window as a budget to be allocated. The same window is shared among these components; removing any of them will affect performance:

A common mistake is stuffing the entire knowledge base into the system prompt. The correct approach is RAG: retrieve first, then only place relevant snippets into the window. Retrieval precision determines answer quality more than context length—this is elaborated in RAG and Retrieval Augmentation.

Prompt Caching: Caching Stable Prefixes

In multi-turn conversations and agent loops, the prefix of each request (tool definitions + system prompt + large chunks of history) is highly repetitive. Prompt caching caches this stable prefix on the server side, and subsequent hits are billed at ~0.1x the price (5-minute TTL is 1.25x, 1-hour TTL is 2x).

All its behaviors can be derived from one invariant:

Prompt caching is prefix matching. If a single byte in the prefix changes, all cache from that position onwards becomes invalid.

The rendering order is tools → system → messages, so cache boundaries should be placed at the junction between "stable" and "volatile":

Practical points:

• Put stable content first, volatile content last. Do not insert now(), UUIDs, user IDs, or random IDs in the system prompt—they are in the prefix and will invalidate everything after them. Place dynamic information towards the end of the messages.
• Breakpoint: Place cache_control: {type: "ephemeral"} (or use top-level auto-caching) on the last block of the stable prefix. There is a maximum of 4 breakpoints per request.
• Minimum cacheable prefix varies by model: Opus 4.8 is 4096 tokens, Sonnet 4.6 / Fable 5 is 2048 tokens—if the prefix is too short, it will silently fail to cache without error.
• Verify hits: Check the response usage.cache_read_input_tokens. If this remains 0 for multiple requests with the same prefix, there is a "silent invalidator" (timestamp in system prompt, unsorted JSON, changing tool set); diff the prefixes of two requests byte-by-byte to find it.

(✅=this layer's cache remains valid). Conclusion: Do not change the tool set or switch models mid-stream—this invalidates the entire cache from the start.

When Conversation Exceeds the Window: Compaction vs Context Editing

As long-running agents accumulate history, it approaches or exceeds the window. Two API primitives work in opposite directions and are often used together:

Remember the difference: compaction is "summarization," context editing is "deletion." Persistent memory across sessions is a third matter (writing key points to files/memory banks), which is not solved within a single window—see Memory and State for details.

Thinking / Effort / Task_Budget: Letting the Model Manage Its Own Budget

Modern Claude (Opus 4.6+) uses adaptive thinking (thinking: {type: "adaptive"})—the model decides when and how deeply to think, rather than manually filling budget_tokens (manual input returns 400 on Opus 4.7/4.8/Fable 5). The master switch for depth is effort:

output_config: { effort: "low" | "medium" | "high" | "xhigh" | "max" }

Lower effort results in fewer, more aggregated tool calls and shorter preambles; high/xhigh are suitable for coding and agents. It directly determines how many tokens are spent on thinking + tools + output in a single round—this is the most direct knob for context budget.

There is also task budget (beta task-budgets-2026-03-13): give the entire agent loop a token budget (minimum 20,000), and the model can see the countdown and self-terminate. It differs from max_tokens: max_tokens is a hard limit for a single response that the model cannot see; task_budget is a "spending suggestion" perceivable by the model.

"Lost in the Middle" and Context Rot: More ≠ Better

The model's attention to the window is uneven: it remembers the beginning and end clearly, but often "cannot see" the middle (referred to as "lost in the middle" in research). Additionally, stuffing too much weak relevant content dilutes the signal—longer context can actually reduce accuracy. This is "context rot."

Two actionable countermeasures:

• Place the most important instructions where the model's attention is strong (earlier in system, or later in user), and use clear separators / XML tags to box key blocks.
• Prefer retrieval over stuffing. Rather than pasting 50 pages of documents and hoping the model finds what it needs, use RAG to hit relevant snippets and place only 2 pages (see RAG and Retrieval Augmentation).

Best Practices

• Treat the window as a budget, not a trash can. Decide what to put in, where to put it, and how to reuse it, rather than blindly stuffing it full—this is the first principle of using any modern model.
• Lock down stable prefixes, place volatile content later. Freeze the system, fix tool order, and put dynamic information (timestamps/IDs/retrieval snippets) towards the back—this is almost the entire secret to cache hits.
• Start with high effort, sweep via eval. Don't reflexively pull to xhigh/max; the relationship is non-monotonic, and high effort is often cheaper on agents.
• Retrieval is better than stuffing. 2 pages from RAG hits beat 50 pages pasted hard (see RAG and Retrieval Augmentation).
• Place key instructions in strong positions + box with separators. Earlier system or later user, separated by XML tags, don't bury them in the middle.
• Proactively manage long conversations, distinguish three things. Compaction summarizes, context editing deletes, Memory persists across sessions—don't use one method to do three things.

Trade-offs and Failure Modes

• Stuffing the window: Assuming more is better triggers context rot and dilutes attention, reducing accuracy → Retrieval is better than stuffing; only place hit snippets.
• Silent cache invalidation: Inserted now()/UUID/unsorted JSON in system prompt, cache_read_input_tokens remains 0 for a long time → Freeze prefix, place dynamic info later, diff requests byte-by-byte to troubleshoot.
• Changing tool set or model within a round: Invalidates the entire prefix cache, multiplying costs → Keep tool set and model constant within the same loop.
• Confusing the two "caches": KV cache is VRAM acceleration within a single generation; prompt caching is prefix reuse across requests → Distinguish layers before optimizing.
• Reflexively maxing out effort: High effort on simple tasks wastes tokens idling → Start with high, sweep via eval (see Reasoning and Thinking).

Frontier: Context as "World"

Pushing the context window to its limits leads to an interesting direction: world models. Alibaba's Qwen team's Qwen-AgentWorld-35B-A3B is a 256K context MoE (35B total params / 3B activated). It is not for chatting, but for simulating the environment the agent is in within the context (terminal, Web, OS...). Such models turn "context" from a "place to stuff data" into a "state承载 an interactive world"—the ultimate goal of context engineering is to treat the window as a state machine to be managed. This line of thought is elaborated in the frontier section of Agent Loops and Tool Use.

References

• Anthropic Official Docs: Prompt Caching, Compaction, Context Editing, Adaptive Thinking & Effort (platform.claude.com, subject to official docs before implementation)
• Local Practice: Local LLM Deployment (KV cache quantization, VRAM trade-offs)
• Research: "Lost in the Middle: How Language Models Use Long Contexts" (Liu et al., 2023)
• Downstream: Agent Loops and Tool Use, MCP and Skills

Keywords: context window, token, tokenizer, KV cache, attention O(n²), prompt caching, prefix match, cache_control, TTL, compaction, context editing, adaptive thinking, effort, task budget, lost in the middle, context rot, RAG, world model