---
title: Context Engineering
url: https://doc.liz6.com/en/ai/01-models-and-context/01-context-engineering
locale: en
area: ai
tags:
- ai
- models-and-context
date: 2026-06-30
modified: 2026-07-16
description: 'Models are stateless—everything they "know" resides in the context window of each request. Treat the window as a budget: cache stable prefixes, compress old history, and load tools on demand, rather than stuffing it indiscriminately. Window management is the primary variable for agent performance.'
---
# Context Engineering
> Models are stateless—everything they "know" resides in the context window of each request. Treat the window as a budget: cache stable prefixes, compress old history, and load tools on demand, rather than stuffing it indiscriminately. Window management is the primary variable for agent performance.
## Overview
Large Language Models (LLMs) are **stateless** within a single request: they do not remember the previous conversation, nor do they have hidden "memory." Their entire understanding of the world is limited to the text provided in that specific request—the **context window**. "Context engineering" is the practice of deciding what to put in this limited, expensive, and position-sensitive window, where to place it, and how to reuse it, in order to achieve more accurate, faster, and cheaper outputs.
This is important because our intuition about LLMs is often misaligned: we assume that "feeding in more data" will always yield better answers. In reality, windows have an upper limit (Claude Opus 4.8 supports 1M tokens), attention computation scales approximately as ~O(n²) with length, and models often struggle to "see clearly" information in the middle of the window. Treating the window as a **budget** to be managed, rather than an bottomless trash can, is the first principle of using any modern model effectively.
Historically, this evolved from "prompt engineering" (how to phrase things): in the era of short, single-turn prompts, phrasing was everything; in the era of long contexts, tool calling, and multi-turn agents, **what is in the window, the order in which it is arranged, and what can be cached** are the dominant factors. Anthropic progressively introduced prompt caching, compaction, context editing, and adaptive thinking as API primitives between 2024 and 2026; these are essentially tools for context engineering.
## The Model Only "Knows" the Context Window in One Round
For every `/v1/messages` request, the model sees a concatenated text block (rendered in the fixed order: `tools → system → messages`):
Key takeaway: The model "remembers" the previous turn only because **the caller sent the entire history again**. The API is stateless; "memory" is maintained by the caller. This also explains why long conversations become increasingly expensive—every turn requires resending the entire history.
## Tokens and Attention: Why Longer is More Expensive and Slower
Text is first tokenized into tokens by the **tokenizer** (Claude's tokens ≠ OpenAI's tokens; do not use `tiktoken` to estimate Claude, as it will underestimate by 15–20%; use `count_tokens` for accuracy). Billing, rate limiting, and window limits are all measured in tokens.
Length brings two types of costs:
- **Compute ~O(n²)**: Self-attention allows each token to attend to all other tokens; doubling the sequence length quadruples the computation. This is the root cause of "longer is slower."
- **KV Cache VRAM**: To avoid recomputing history for every generated token, inference engines cache the Key/Value tensors for each layer of the history. VRAM usage grows linearly with context length. This is especially tight for local deployments—see [Local LLM Deployment](/homelab/local-llm.md): a 64k context f16 KV cache requires ~5.37 GB, which won't fit on a 24GB card, requiring quantization to Q8_0 to reduce it to ~2.85 GB. Cloud APIs don't expose this VRAM, but it is the physical reason why "long context is more expensive."
> KV cache (tensors cached by the inference engine for acceleration) and **prompt caching** (prefix caching by the API to save money) discussed below are two different layers of "caching." Do not confuse them. The former accelerates within a single generation; the latter reuses prefixes across requests.
## Context Budget: Who is Competing for this Window
Treat the window as a budget to be allocated. The same window is shared among these components; removing any of them will affect performance:
| Consumer | Typical Size | Management Strategy |
|--------|----------|----------|
| Tool definitions (tools) | Tens to thousands of tokens/tool | Use tool search to load on demand when there are many tools; don't dump them all at once. |
| System prompt (system) | Fixed | Freeze it (see caching); don't insert timestamps inside it. |
| History (messages) | Grows linearly with conversation | Compaction / context editing |
| Retrieved Content (RAG) | Controllable | Retrieval is better than stuffing: only include hit snippets, don't dump the whole database. |
| Output (output) | Reserved max_tokens | Streaming + reasonable max_tokens |
| Thinking | Determined by effort | Adjust via effort, don't stuff manually. |
A common mistake is stuffing the entire knowledge base into the system prompt. The correct approach is **RAG**: retrieve first, then only place relevant snippets into the window. Retrieval precision determines answer quality more than context length—this is elaborated in [RAG and Retrieval Augmentation](/ai/03-applications-and-production/01-rag-and-retrieval-augmented-generation.md).
## Prompt Caching: Caching Stable Prefixes
In multi-turn conversations and agent loops, the prefix of each request (tool definitions + system prompt + large chunks of history) is highly repetitive. **Prompt caching** caches this stable prefix on the server side, and subsequent hits are billed at ~0.1x the price (5-minute TTL is 1.25x, 1-hour TTL is 2x).
All its behaviors can be derived from one invariant:
> **Prompt caching is prefix matching. If a single byte in the prefix changes, all cache from that position onwards becomes invalid.**
The rendering order is `tools → system → messages`, so cache boundaries should be placed at the junction between "stable" and "volatile":
Practical points:
- **Put stable content first, volatile content last.** Do **not** insert `now()`, UUIDs, user IDs, or random IDs in the system prompt—they are in the prefix and will invalidate everything after them. Place dynamic information towards the end of the messages.
- **Breakpoint**: Place `cache_control: {type: "ephemeral"}` (or use top-level auto-caching) on the last block of the stable prefix. There is a maximum of 4 breakpoints per request.
- **Minimum cacheable prefix** varies by model: Opus 4.8 is 4096 tokens, Sonnet 4.6 / Fable 5 is 2048 tokens—if the prefix is too short, it will **silently fail to cache** without error.
- **Verify hits**: Check the response `usage.cache_read_input_tokens`. If this remains 0 for multiple requests with the same prefix, there is a "silent invalidator" (timestamp in system prompt, unsorted JSON, changing tool set); diff the prefixes of two requests byte-by-byte to find it.
| What Changed | Tool Cache | System Cache | Message Cache |
|----------|:--:|:--:|:--:|
| Tool definitions (add/remove/reorder), change model | ❌ | ❌ | ❌ |
| System prompt content | ✅ Retained | ❌ | ❌ |
| Message content, `tool_choice`, toggle thinking | ✅ | ✅ | ❌ |
(✅=this layer's cache remains valid). Conclusion: **Do not change the tool set or switch models mid-stream**—this invalidates the entire cache from the start.
## When Conversation Exceeds the Window: Compaction vs Context Editing
As long-running agents accumulate history, it approaches or exceeds the window. Two API primitives work in opposite directions and are often used together:
| | Compaction | Context Editing |
|---|---|---|
| What it does | **Summarizes** early history into a compaction block | **Clears** old tool_result / thinking |
| Information | Summarizes and retains | Deletes directly |
| When to use | Approaching window limit, need to continue conversation | Old tool outputs are irrelevant, want to slim down |
| Key Pitfall | Must pass the `response.content` (including compaction block) **entirely back**; passing only text will lose state | Only deletes tool_use/thinking, does not alter conversation structure |
| Beta Header | `compact-2026-01-12` | `context-management-2025-06-27` |
Remember the difference: compaction is "summarization," context editing is "deletion." Persistent memory across sessions is a third matter (writing key points to files/memory banks), which is not solved within a single window—see [Memory and State](/ai/02-Agent/03-memory-and-state.md) for details.
## Thinking / Effort / Task_Budget: Letting the Model Manage Its Own Budget
Modern Claude (Opus 4.6+) uses **adaptive thinking** (`thinking: {type: "adaptive"}`)—the model decides when and how deeply to think, rather than manually filling `budget_tokens` (manual input returns 400 on Opus 4.7/4.8/Fable 5). The master switch for depth is **effort**:
```
output_config: { effort: "low" | "medium" | "high" | "xhigh" | "max" }
```
Lower `effort` results in fewer, more aggregated tool calls and shorter preambles; `high`/`xhigh` are suitable for coding and agents. It directly determines how many tokens are spent on thinking + tools + output in a single round—this is the most direct knob for context budget.
There is also **task budget** (beta `task-budgets-2026-03-13`): give the entire agent loop a token budget (minimum 20,000), and the model can **see the countdown** and self-terminate. It differs from `max_tokens`: `max_tokens` is a hard limit for a single response that the model cannot see; `task_budget` is a "spending suggestion" perceivable by the model.
## "Lost in the Middle" and Context Rot: More ≠ Better
The model's attention to the window is **uneven**: it remembers the beginning and end clearly, but often "cannot see" the middle (referred to as "lost in the middle" in research). Additionally, stuffing too much weak relevant content dilutes the signal—longer context can actually **reduce** accuracy. This is "context rot."
Two actionable countermeasures:
- **Place the most important instructions where the model's attention is strong** (earlier in system, or later in user), and use clear separators / XML tags to box key blocks.
- **Prefer retrieval over stuffing.** Rather than pasting 50 pages of documents and hoping the model finds what it needs, use RAG to hit relevant snippets and place only 2 pages (see [RAG and Retrieval Augmentation](/ai/03-applications-and-production/01-rag-and-retrieval-augmented-generation.md)).
## Best Practices
- **Treat the window as a budget, not a trash can.** Decide what to put in, where to put it, and how to reuse it, rather than blindly stuffing it full—this is the first principle of using any modern model.
- **Lock down stable prefixes, place volatile content later.** Freeze the system, fix tool order, and put dynamic information (timestamps/IDs/retrieval snippets) towards the back—this is almost the entire secret to cache hits.
- **Start with `high` effort, sweep via eval.** Don't reflexively pull to `xhigh`/`max`; the relationship is non-monotonic, and high effort is often cheaper on agents.
- **Retrieval is better than stuffing.** 2 pages from RAG hits beat 50 pages pasted hard (see [RAG and Retrieval Augmentation](/ai/03-applications-and-production/01-rag-and-retrieval-augmented-generation.md)).
- **Place key instructions in strong positions + box with separators.** Earlier system or later user, separated by XML tags, don't bury them in the middle.
- **Proactively manage long conversations, distinguish three things.** Compaction summarizes, context editing deletes, [Memory](/ai/02-Agent/03-memory-and-state.md) persists across sessions—don't use one method to do three things.
## Trade-offs and Failure Modes
- **Stuffing the window**: Assuming more is better triggers context rot and dilutes attention, reducing accuracy → Retrieval is better than stuffing; only place hit snippets.
- **Silent cache invalidation**: Inserted `now()`/UUID/unsorted JSON in system prompt, `cache_read_input_tokens` remains 0 for a long time → Freeze prefix, place dynamic info later, diff requests byte-by-byte to troubleshoot.
- **Changing tool set or model within a round**: Invalidates the entire prefix cache, multiplying costs → Keep tool set and model constant within the same loop.
- **Confusing the two "caches"**: KV cache is VRAM acceleration within a single generation; prompt caching is prefix reuse across requests → Distinguish layers before optimizing.
- **Reflexively maxing out effort**: High effort on simple tasks wastes tokens idling → Start with `high`, sweep via eval (see [Reasoning and Thinking](/ai/01-models-and-context/04-reasoning-and-thinking.md)).
## Frontier: Context as "World"
Pushing the context window to its limits leads to an interesting direction: **world models**. Alibaba's Qwen team's [Qwen-AgentWorld-35B-A3B](https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B) is a 256K context MoE (35B total params / 3B activated). It is not for chatting, but for **simulating the environment the agent is in within the context** (terminal, Web, OS...). Such models turn "context" from a "place to stuff data" into a "state承载 an interactive world"—the ultimate goal of context engineering is to treat the window as a state machine to be managed. This line of thought is elaborated in the frontier section of [Agent Loops and Tool Use](/ai/02-Agent/01-the-agent-loop-and-tool-use.md).
## References
- **Anthropic Official Docs**: Prompt Caching, Compaction, Context Editing, Adaptive Thinking & Effort (platform.claude.com, subject to official docs before implementation)
- **Local Practice**: [Local LLM Deployment](/homelab/local-llm.md) (KV cache quantization, VRAM trade-offs)
- **Research**: "Lost in the Middle: How Language Models Use Long Contexts" (Liu et al., 2023)
- **Downstream**: [Agent Loops and Tool Use](/ai/02-Agent/01-the-agent-loop-and-tool-use.md), [MCP and Skills](/ai/02-Agent/02-mcp-and-skills.md)
*Keywords: context window, token, tokenizer, KV cache, attention O(n²), prompt caching, prefix match, cache_control, TTL, compaction, context editing, adaptive thinking, effort, task budget, lost in the middle, context rot, RAG, world model*