RAG and Retrieval-Augmented Generation

Don’t stuff the entire library into the window and hope the model finds what it needs. RAG breaks "infinite knowledge" into "retrieve first, then inject the few matching pages into the context"—the quality of the answer depends not on how long the context is, but on how accurate the retrieval is.

Overview

Context Engineering repeatedly emphasizes one rule: retrieval beats stuffing. This article expands that premise into a practical engineering approach: RAG (retrieval-augmented generation).

A model’s parameters contain the knowledge it saw during training, but it does not know about private documents, today’s data, or the conventions of this project. Moreover, the context window is limited, and attention mechanisms struggle with the middle section (lost in the middle). There are two dead ends: stuffing the entire knowledge base into the system prompt (blows up the window, dilutes signals, and is expensive), or hoping that fine-tuning welds the knowledge into the weights (slow, expensive, and hard to update). RAG is the third path: keep the knowledge in a retrieval store outside the window, and only retrieve the few fragments relevant to the current question, appending them to the context.

In one sentence, the essence is: RAG transforms the question of "how much the model knows" into "whether the retrieval system can rank the correct fragments at the top." Thus, the bottleneck for answer quality shifts from the model to retrieval—which is why most of this article focuses on how to retrieve accurately, rather than how to generate.

Core Loop: retrieve → augment → generate

flowchart LR
    Q["User Question"] --> EMB["Query Vectorization"]
    EMB --> SEARCH["Vector Store Retrieval<br/>top-k similar fragments"]
    DOCS["Knowledge Base<br/>(chunked + pre-embedded)"] -.Offline Indexing.-> SEARCH
    SEARCH --> RERANK["Rerank<br/>(optional, fine-ranking)"]
    RERANK --> AUG["Append to Context<br/>Matched Fragments + Question"]
    AUG --> LLM["Model Generation<br/>(with citations)"]
    LLM --> A["Answer"]

Divided into two phases: offline indexing (chunking documents, vectorizing, and loading them into the vector store, done once) and online retrieval and generation (vectorizing the question, retrieving, concatenating, and generating for each request). The latter connects directly to the Agent Loop—retrieval can be a tool, allowing the model to decide when and what to search.

Embeddings and Vector Retrieval: Why "Semantic" Search Works

Keyword retrieval matches literally; asking "how to save VRAM on a GPU" won’t find a paragraph discussing "KV cache quantization." Embeddings map a piece of text into a high-dimensional vector, so semantically similar texts have similar vectors. Retrieval then becomes "finding the nearest neighbors in vector space"—searching by meaning, not by literal characters.

• Similarity: Cosine similarity is commonly used. Pre-embed all fragments in the store. When querying, only embed the query once, calculate its similarity with the store vectors, and take the top-k.
• Local Execution: Embeddings don’t need to be cloud-based. llama.cpp can run embedding models directly (e.g., Qwen3-Embedding, bge-m3). Paired with a local vector database (sqlite-vec, Qdrant, Chroma), this forms an offline RAG system—see Local LLM Deployment. Embedding models are small, batch processing is fast, and VRAM pressure is far lower than for generation models.
• Dimensions and Scale: For thousands to tens of thousands of fragments, brute-force nearest neighbor search is sufficient. For millions, use ANN indexes (HNSW, IVF) to trade accuracy for speed.

Embedding models ≠ Generation models; they have separate tokenizers and are not interchangeable. The model used to embed the knowledge base and the model used to embed the query must be the same—switching models requires re-embedding the entire store, otherwise vectors won’t be in the same space, and retrieval will fail completely.

Chunking Strategy: If Chunking Fails, Even Perfect Retrieval is Useless

Documents must be cut into chunks before embedding. Chunking is the most underestimated yet most impactful step in RAG:

Rule of thumb: First split according to the document’s natural structure (this site uses ## sections), then fall back to character-count-based splitting for overly large chunks. Chunking quality is far more important than the choice of vector database.

Improving Hit Rates: Hybrid Search and Reranking

Pure vector retrieval has blind spots: exact proper nouns, error codes, and IDs are often better matched literally. Two methods to improve accuracy:

• Hybrid Search: Combine vector retrieval (handles semantics) with keyword retrieval BM25 (handles literals), fusing results from both (e.g., RRF, Reciprocal Rank Fusion). When asking "what is model_context_window_exceeded?", BM25 can precisely match that string, whereas vector search might not.
• Reranking: First, retrieve a coarse top-50. Then, use a cross-encoder reranker to score each query-fragment pair precisely, selecting the truly most relevant top-5 to append to the window. Recall ensures "nothing is missed"; reranking ensures "nothing irrelevant is included." This two-stage approach is more window-efficient and accurate than simply increasing k.

Don’t rely on increasing k as a safety net. The larger k is, the more noise enters the window, triggering context rot and wasting tokens. It’s better to have broad recall and strict reranking, finally placing only a few high-quality pages.

Appending to the Window: Reusing Context Engineering

Once fragments are retrieved, how you place them in the context directly reuses all conclusions from Context Engineering:

• Position: Place matched fragments in high-attention positions (after the system prompt at the beginning, or immediately before the question). Use clear separators or XML tags to frame them; don’t bury them in the middle.
• Caching: Put stable instructions and a small amount of unchanging reference material in the prefix (hitting prompt caching). Place the variable retrieval fragments at the end, not inserted into the prefix, to avoid breaking the cache.
• Require Citations: Ask the model to annotate which fragment each conclusion comes from in the answer (using chunk metadata). Claude’s citations capability allows cited fragments to carry location information, but note that it is mutually exclusive with structured output output_config.format; choose one.

Evaluating Retrieval: Quantifying "Accuracy"

RAG failures often lie not in generation, but in retrieval—the model answers incorrectly often because the correct fragment was never retrieved. Therefore, evaluate the two stages separately (refer to Evaluation and Observability):

• Retrieval Quality (Programmatic, Most Reliable): For a set of questions with "standard answer sources," measure recall@k (did the correct fragment enter the top-k?), precision@k (how many in the top-k are truly relevant?), and MRR. This step relies purely on code assertions, is zero-cost, and unbiased.
• Generation Quality (Open-ended): Whether the answer is faithful to the retrieved content, free of hallucinations, and has correct citations—use LLM-as-judge, keeping the judge separate from the subject.

To locate problems, look at retrieval first: low recall means go back to adjust chunking, change embeddings, or add reranking; high recall but wrong answers indicate generation or prompt issues.

RAG vs. Memory vs. Long Context: Don’t Confuse Them

Three commonly confused methods for "making the model know more" have clear boundaries (see Memory and State):

RAG retrieves "objective knowledge fragments," while memory stores "subjective experience and conclusions." RAG re-retrieves every turn; memory is actively written and read by the model. Long-running agents often use all three in combination.

Best Practices

• Measure retrieval first, then tune generation. Build a small evaluation set with "source-standard answers," and maximize recall@k first—if retrieval is inaccurate, changing models or tuning prompts is like scratching an itch from outside the boot.
• Chunk by semantic structure, with sufficient metadata. Prioritize titles/paragraphs/function boundaries over fixed character counts; attach source + chapter to each chunk for both filtering and citation.
• Broad recall, strict reranking, minimal context. Coarse recall top-50 → rerank → place only top-3~5; don’t force-feed by increasing k.
• Use the same embedding model for queries and the store. Switching models requires re-embedding the entire store; upgrade embeddings via canary releases, don’t replace them in place.
• Use hybrid search for proper nouns/error codes/IDs. Pure vector search has blind spots for literal exact matches; add a BM25 path.
• Always place retrieval fragments at the end of the window. Protect prefix caching; stable instructions go first, variable hits go last.
• Make answers citable and traceable. Use chunk metadata to annotate sources, reducing hallucinations and facilitating manual verification.

Trade-offs and Failure Modes

• Stuffing the entire library into the system: Blows up the window, dilutes signals, expensive → Only place matched fragments after retrieval.
• Chunks too large/too small: Large chunks dilute vectors; small chunks break semantics → Split at semantic boundaries + moderate overlap, tune recall via testing.
• Inconsistent query and store embeddings: Vectors in different spaces, retrieval fails → Lock to one embedding model; rebuild index when switching.
• Pure vector search misses exact items: Error codes/proper nouns fail literal matching → Hybrid search with BM25.
• Relying on large k as a safety net: Noise enters window, triggers context rot, wastes tokens → Broad recall + reranking + minimal placement.
• Focusing only on generation, ignoring retrieval: Blaming the model for wrong answers when fragments weren’t recalled → Measure recall@k first, then attribute causes.
• Inserting retrieval fragments into the cache prefix: Hits vary per turn, invalidating prefix cache → Place hits at the end, keep prefix stable.

References

• Research: "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (Lewis et al., 2020), "Lost in the Middle" (Liu et al., 2023)
• Anthropic Official Docs: Citations, Structured Outputs (platform.claude.com, subject to official documentation)
• Local Practice: Local LLM Deployment (run embeddings and vector store locally)
• Upstream/Downstream: Context Engineering, Memory and State, Evaluation and Observability

Keywords: RAG, retrieval-augmented generation, embedding, vector retrieval, cosine similarity, nearest neighbor, ANN, HNSW, chunk, chunking, overlap, chunking, BM25, hybrid search, RRF, rerank, cross-encoder, recall@k, precision@k, MRR, citations, hallucination, retrieval beats stuffing, vector database, sqlite-vec, Qdrant