Retrieval-Augmented Generation¶

Summary — Give the model the facts it needs at inference time instead of hoping they were memorized during training. RAG is the standard architecture for grounding LLMs in private or current data. This page covers the pipeline stage by stage, the chunking decisions that dominate quality, a failure taxonomy that tells you which stage to fix, and how to evaluate each stage separately.

Prerequisites: → Embeddings & vector search · Next: → Agents & tool use

1. The pipeline¶

  INDEXING (offline, once)
  ────────────────────────
  documents ──► parse ──► chunk ──► embed ──► vector index
                                      │
                                      └──► also: BM25 index, metadata store

  QUERYING (online, per request)
  ──────────────────────────────
  user query ──► [rewrite] ──► retrieve (hybrid) ──► rerank ──► top-k chunks
                                                                    │
                                                                    ▼
                          LLM ◄── prompt = system + chunks + query
                           │
                           ▼
                        answer + citations

Why RAG rather than fine-tuning

Fine-tuning changes behaviour; retrieval changes knowledge. Fine-tuning a model on your documents teaches it their style and makes it hallucinate confidently in that style. Retrieval puts the actual text in front of it.

	RAG	Fine-tuning
Update knowledge	re-index (minutes)	retrain (days)
Attribution	✅ natural — cite the chunk	❌ impossible
Access control	✅ filter at retrieval time	❌ baked into weights
Cost per query	higher (more tokens)	lower
Teaches format/style	❌	✅
Hallucination risk	lower (grounded)	higher

2. Chunking: the decision that matters most¶

That comparison assumes the retrieved chunks are actually good ones. Whether they are depends almost entirely on the first real decision in building a RAG system: how the source documents get split up in the first place.

Chunking quality sets the ceiling for everything downstream

No amount of retrieval or reranking sophistication recovers from chunks that split a table in half or separate a claim from its qualifier.

   TOO SMALL (128 tokens)            TOO LARGE (2000 tokens)

   ┌──────────┐                      ┌────────────────────────┐
   │ "...the  │  ← context lost      │ topic A                │
   └──────────┘                      │ topic B   ← the query  │
   ┌──────────┐                      │ topic C     only needs │
   │ rate is  │  ← dangling          │ topic D     topic B    │
   └──────────┘                      │ topic E                │
   ┌──────────┐                      └────────────────────────┘
   │ 4.5%..." │                      embedding is a blurred average;
   └──────────┘                      retrieval precision drops

Strategies, in increasing order of quality and effort:

Strategy	How	When
Fixed-size	$N$ tokens, $M$ overlap	baseline; always works
Recursive	split on `\n\n` → `\n` → `.` → until under the limit	⭐ good default
Structural	split on Markdown headings, HTML sections, code functions	⭐ best when structure exists
Semantic	split where consecutive-sentence embedding similarity drops	expensive, modest gain
Proposition	an LLM rewrites text into standalone factual statements	expensive, high quality
Late chunking	embed the whole document, then pool per chunk	keeps global context in each chunk embedding

Starting parameters: 512 tokens with 50–100 tokens of overlap. Overlap prevents a fact from being severed at a boundary.

Three techniques that punch above their weight

(a) Contextual retrieval — prepend an LLM-generated context sentence to each chunk before embedding it:

  Original chunk:  "Revenue grew 3% over the prior quarter."
  Contextualized:  "This chunk is from ACME Corp's Q2 2024 10-Q, discussing
                    segment performance. Revenue grew 3% over the prior quarter."

Anthropic reported this substantially reduces retrieval failures. It costs one cheap LLM call per chunk at index time — a one-off expense for a permanent gain. It directly fixes the "this chunk is meaningless without knowing what document it came from" problem.

(b) Parent-document retrieval — embed small chunks for precision, but return their larger parent section to the LLM. Best of both: sharp retrieval, complete context.

(c) Multi-representation indexing — index a summary or hypothetical question for each chunk, but return the chunk itself. Queries look more like questions than like document prose, so matching question-to-question works better than question-to-prose.

3. Query processing¶

Chunking decides what's in the index. A separate lever is what you actually search with — the raw user query is often a worse search string than something derived from it.

The user's query is often a poor retrieval key.

Technique	Problem it solves
Query rewriting	conversational references ("what about last year?" → a standalone query)
Multi-query	generate 3–5 paraphrases, retrieve for each, union the results
HyDE	generate a hypothetical answer, embed that, and retrieve with it
Decomposition	split a multi-hop question into sub-questions
Step-back	ask a more general question first, retrieve background, then the specific one
Routing	choose which index/collection to search

HyDE (Hypothetical Document Embeddings) is the cleverest of these

Queries and documents live in different linguistic registers — a question ("what causes X?") looks nothing like an answer passage ("X is caused by..."). Asking the LLM to hallucinate a plausible answer and embedding that puts the search vector in document-space, where the real answers are. The hallucination's factual content is irrelevant; only its shape matters.

Pitfall

Each of these adds latency and cost. Multi-query with 5 variants means 5× the retrieval work plus an LLM call. Measure whether it helps on your data before adopting it.

4. The failure taxonomy¶

Chunking, indexing and query processing are all places a RAG pipeline can go wrong, each in a different way. Rather than guess which one is broken, it helps to have a structured list of every failure mode and where in the pipeline each one points.

This table is the most practically useful thing on this page. When RAG gives a bad answer, diagnose which stage failed:

#	Failure	Symptom	Fix
1	Missing content	the answer isn't in the corpus at all	ingest more; detect and say "I don't know"
2	Missed the top-k	the right chunk exists but wasn't retrieved	hybrid search, better embeddings, larger $k$, reranker
3	Lost in reranking/context	retrieved but not used	reorder (best first/last), fewer chunks, reranker
4	Not extracted	the chunk is in context but the LLM missed it	better prompt, stronger model, less distraction
5	Wrong format	right facts, wrong shape	explicit format instruction, structured output
6	Incomplete	partial answer	multi-hop retrieval, larger $k$
7	Ignored context	model answers from parametric memory instead	prompt to use only the context; check for conflicts
8	Bad chunk boundaries	truncated tables, split claims	structural chunking, larger overlap

The diagnostic procedure

For a failing query, check in order — (a) is the answer in the corpus? (b) is the right chunk in the top-50? (c) is it in the top-5 after reranking? (d) did the LLM use it? Each "no" points at exactly one fix. Debugging RAG without this decomposition is guesswork.

Empirically the most common failure is #2 — retrieval recall. This is why hybrid search and reranking are the two highest-value additions.

5. Generation¶

Retrieval failures dominate, but they aren't the only ones — a query can retrieve the perfect chunk and the system can still produce a bad answer, if the generation step mishandles what it was given.

A prompt template that handles the things that actually go wrong:

RAG_PROMPT = """Answer the question using ONLY the provided context.

Rules:
- If the context does not contain the answer, say "I don't have enough information
  to answer that." Do not use outside knowledge.
- Cite the source id in brackets after each claim, like [3].
- If sources conflict, say so and present both.
- Quote exact figures and dates rather than paraphrasing them.

<context>
{context}
</context>

Question: {question}

Answer:"""

def format_context(chunks):
    # Put the most relevant chunk FIRST and the second-most LAST:
    # both ends of the context are attended to more strongly.
    if len(chunks) > 2:
        chunks = [chunks[0]] + chunks[2:] + [chunks[1]]
    return "\n\n".join(
        f"[{i}] (source: {c['source']}, section: {c.get('section','?')})\n{c['text']}"
        for i, c in enumerate(chunks, 1))

The reordering trick exploits 'lost in the middle'

(→ Long context §6). Put your two best chunks at the extremes, where attention is strongest.

Knowledge conflict

When the retrieved context contradicts the model's parametric knowledge, behaviour is unpredictable. Models generally favour the context when it is clear and specific, and fall back on parametric memory when the context is vague. Explicit instructions help but are not a guarantee. For high-stakes applications, verify that cited claims actually appear in the cited chunk.

6. Evaluation¶

That knowledge-conflict problem is exactly the kind of failure a RAG system needs to be tested for, not just designed against. Turning "did the model use the context correctly" into a number you can track is the next step.

Evaluate retrieval and generation separately

A single end-to-end score tells you something is wrong but not what.

Retrieval metrics¶

Metric	Formula	Meaning
Recall@k	(relevant in top-$k$) / (total relevant)	did we find it? ⭐ the one that matters most
Precision@k	(relevant in top-$k$) / $k$	how much noise?
MRR	$\frac{1}{\\|Q\\|}\sum \frac{1}{\text{rank of first relevant}}$	how high is the first hit?
nDCG@k	$\frac{\text{DCG}@k}{\text{IDCG}@k}$, $\text{DCG}=\sum_i \frac{rel_i}{\log_2(i+1)}$	graded relevance, position-weighted
Hit rate	fraction of queries with ≥1 relevant hit	coarse but interpretable

nDCG example. Retrieved relevance grades $[3, 0, 2, 1]$ (ideal would be $[3,2,1,0]$):

\[\text{DCG} = \frac{3}{\log_2 2} + \frac{0}{\log_2 3} + \frac{2}{\log_2 4} + \frac{1}{\log_2 5} = 3 + 0 + 1 + 0.431 = 4.431$$ $$\text{IDCG} = 3 + \frac{2}{1.585} + \frac{1}{2} + 0 = 3 + 1.262 + 0.5 = 4.762$$ $$\text{nDCG} = 4.431/4.762 = \mathbf{0.931}\]

Generation metrics (the RAGAS framework)¶

Metric	Question it answers	How
Faithfulness	are all claims supported by the context?	decompose into claims, verify each against the context
Answer relevance	does it address the question?	generate questions from the answer, compare to the original
Context precision	is the retrieved context on-topic?	judge each chunk's relevance
Context recall	does the context contain everything needed?	check the ground-truth answer's claims against the context

Faithfulness is the metric to prioritize

It directly measures hallucination, which is the failure mode RAG exists to prevent. A system that is 95% faithful and 80% relevant is far more trustworthy than the reverse.

Building an evaluation set without human labels: have an LLM generate questions from your chunks. The source chunk is then the ground-truth relevant document by construction. This gives you a few hundred retrieval test cases in an hour. ⚠️ It is biased toward questions that are easy to answer from a single chunk — supplement with real user queries as soon as you have them.

7. Advanced patterns¶

Faithfulness and context recall tell you the plain pipeline from §1 is working. For harder queries — ones that need several documents, or a query that has to be broken into parts — that plain pipeline isn't enough, and a few patterns extend it.

Pattern	Idea	When
Self-RAG	the model decides whether to retrieve, and critiques what it gets	mixed query types
Corrective RAG (CRAG)	grade retrieved docs; if poor, fall back to web search	open-domain
GraphRAG	build an entity/relation graph; retrieve subgraphs and community summaries	questions spanning many documents
Agentic RAG	the LLM issues retrieval as a tool call, iteratively	complex multi-hop research
Recursive summarization	hierarchical summaries (RAPTOR)	"what is this whole corpus about?"
Long-context RAG	retrieve 50 chunks instead of 5	when the window allows it

GraphRAG addresses a genuine structural gap

Standard RAG answers "what does document X say about Y?" It cannot answer "what are the main themes across all 10,000 documents?" — no single chunk contains that, so retrieval has nothing to find. GraphRAG precomputes community summaries at index time so global questions have something to retrieve.

Long-context RAG is underrated

Retrieval recall failures dominate (§4), and a big context window lets you trade precision for recall — retrieve 50 chunks and let the model sort it out. This is more robust than heroic efforts to get $k=5$ exactly right. → Long context §7

8. A complete implementation¶

from dataclasses import dataclass

@dataclass
class Chunk:
    text: str
    source: str
    section: str = ""
    parent_id: str | None = None

class RAGPipeline:
    def __init__(self, retriever, reranker, llm):
        self.retriever, self.reranker, self.llm = retriever, reranker, llm

    def index(self, documents, chunk_size=512, overlap=64):
        chunks = []
        for doc in documents:
            for c in recursive_split(doc.text, chunk_size, overlap):
                # contextual retrieval: a one-off LLM call per chunk at index time
                ctx = self.llm(
                    f"<document>{doc.text[:4000]}</document>\n"
                    f"<chunk>{c}</chunk>\n"
                    "In one sentence, situate this chunk within the document. "
                    "Output only that sentence.")
                chunks.append(Chunk(text=f"{ctx}\n\n{c}", source=doc.id,
                                    section=doc.section))
        self.retriever.build(chunks)      # builds BOTH the dense and BM25 indexes
        self.chunks = chunks

    def query(self, question, k=5, candidates=50, filters=None):
        # 1. retrieve widely (hybrid), with metadata filters for access control
        cands = self.retriever.search(question, n=candidates, filters=filters)
        # 2. rerank with a cross-encoder
        scored = self.reranker.rank(question, cands)
        top = scored[:k]
        # 3. abstain rather than answer from irrelevant context
        if not top or top[0].score < RELEVANCE_FLOOR:
            return {"answer": "I don't have enough information to answer that.",
                    "sources": []}
        # 4. generate with citations
        answer = self.llm(RAG_PROMPT.format(
            context=format_context(top), question=question))
        return {"answer": answer,
                "sources": [{"id": i, "source": c.source, "text": c.text[:200]}
                            for i, c in enumerate(top, 1)]}

Pitfall

The RELEVANCE_FLOOR check is the single most important line for trustworthiness. Without it, a query with no good match still retrieves the five least bad chunks and the model dutifully answers from irrelevant context. Calibrate the floor on known-unanswerable queries.

Metadata filters are how you do access control

Never rely on the LLM to withhold information present in its context — filter at retrieval time, before anything reaches the model.

9. Exercises¶

Problem 1 — the failure taxonomy, diagnosed. A RAG system correctly retrieves the right chunk into the top-5, and it survives reranking into the top-3, but the final answer still omits a key fact that was in the retrieved context. Using §4's 8-point taxonomy, which failure number does this match, and what's the recommended fix?

Solution

This matches failure #4, "Not extracted": "the chunk is in context but the LLM missed it." Retrieval (failures #1–2) and reranking/ordering (failure #3) are both ruled out by the problem statement (the chunk made it to top-3, presumably within the context sent to the model). Per §4's fix column: "better prompt, stronger model, less distraction" — concretely, check whether the context has too many other chunks competing for attention (per the → Long context "lost in the middle" effect), try moving the critical chunk to the start or end of the context (§5's reordering trick), or make the prompt more explicit about needing to use all provided facts, not just the most salient one.

Problem 2 — contextual retrieval's cost, worked. A corpus has 50,000 chunks. Using §2's contextual-retrieval technique (one LLM call per chunk at index time to generate a situating sentence), and assuming each call costs $0.0003 (a small/cheap model) and takes 0.5 seconds (assume full parallelism across many concurrent calls, so wall-clock isn't simply $50000\times0.5$s), what's the one-time indexing cost in dollars? Per §2, is this a recurring cost or a one-off?

Solution

$50{,}000\times\$0.0003=\$15.00$ — a modest one-time cost.

Per §2: "it costs one cheap LLM call per chunk at index time — a one-off expense for a permanent gain." This is not recurring for existing chunks — it only needs to run once per chunk, at the time that chunk is first indexed. It does recur incrementally as new documents are added to the corpus (each new chunk needs its own contextualizing call), but re-indexing the same 50,000 chunks repeatedly would be unnecessary and wasteful — the cost structure is "pay once per chunk, forever," not "pay per query" or "pay per re-index."

Problem 3 — abstention floor, calibrating it. Using §8's RELEVANCE_FLOOR mechanism, suppose you test your system on 30 deliberately-unanswerable queries (questions with no answer in the corpus) and 30 answerable ones, and plot the reranker's top-1 score for each group. The unanswerable group's scores cluster around 0.15–0.35; the answerable group clusters around 0.55–0.95, with some overlap around 0.4–0.5. Where would you set RELEVANCE_FLOOR, and what's the trade-off at each end of that overlap range?

Solution

Somewhere in the 0.4–0.5 overlap band is the right zone — exactly matching §8's instruction to "calibrate the floor on known-unanswerable queries."

Setting the floor near 0.4 (the low end of the overlap): fewer answerable queries get wrongly abstained-on (higher recall of real answers), but more unanswerable queries slip through and get a hallucinated-sounding "answer" from irrelevant context (worse precision on abstention) — riskier for trustworthiness.

Setting the floor near 0.5 (the high end): more unanswerable queries correctly trigger "I don't have enough information" (safer), but some genuinely answerable queries with weaker retrieval scores get incorrectly abstained-on too (lower recall, more frustrating false negatives for real users). Per §8's framing of the floor as "the single most important line for trustworthiness," the right choice depends on which failure mode is more costly in your specific application — a medical or legal RAG system would likely lean toward 0.5 (favor safety over recall), while a casual internal search tool might lean toward 0.4.

10. Key takeaways¶

#	Takeaway
1	RAG supplies knowledge; fine-tuning shapes behaviour. Use the right tool.
2	Chunking sets the ceiling. Start at 512 tokens with overlap, and respect document structure.
3	Contextual retrieval (prepend an LLM-written situating sentence) is a cheap, large win.
4	HyDE works by moving the query into document-space; the hallucinated answer's accuracy is irrelevant.
5	Use the 8-way failure taxonomy to identify which stage broke before changing anything.
6	Retrieval recall is the most common failure → hybrid search + reranking are the top two fixes.
7	Put the best chunks at the start and end of the context.
8	Evaluate retrieval (recall@k, nDCG) and generation (faithfulness) separately.
9	Always implement an abstention path with a relevance floor.
10	Enforce access control with retrieval-time metadata filters, never by instructing the model.

Strategy	How	When
Fixed-size	\(N\) tokens, \(M\) overlap	baseline; always works
Recursive	split on `\n\n` → `\n` → `.` → until under the limit	⭐ good default
Structural	split on Markdown headings, HTML sections, code functions	⭐ best when structure exists
Semantic	split where consecutive-sentence embedding similarity drops	expensive, modest gain
Proposition	an LLM rewrites text into standalone factual statements	expensive, high quality
Late chunking	embed the whole document, then pool per chunk	keeps global context in each chunk embedding

Metric	Formula	Meaning
Recall@k	(relevant in top-\(k\)) / (total relevant)	did we find it? ⭐ the one that matters most
Precision@k	(relevant in top-\(k\)) / \(k\)	how much noise?
MRR	\(\frac{1}{\\|Q\\|}\sum \frac{1}{\text{rank of first relevant}}\)	how high is the first hit?
nDCG@k	\(\frac{\text{DCG}@k}{\text{IDCG}@k}\), \(\text{DCG}=\sum_i \frac{rel_i}{\log_2(i+1)}\)	graded relevance, position-weighted
Hit rate	fraction of queries with ≥1 relevant hit	coarse but interpretable

Retrieval-Augmented Generation¶

1. The pipeline¶

2. Chunking: the decision that matters most¶

3. Query processing¶

4. The failure taxonomy¶

5. Generation¶

6. Evaluation¶

Retrieval metrics¶

Generation metrics (the RAGAS framework)¶

7. Advanced patterns¶

8. A complete implementation¶

9. Exercises¶

10. Key takeaways¶

Further reading¶