The 8 Layers of the RAG Pipeline: Why Retrieval, Not Your Model, Decides Answer Quality
Every mediocre RAG system gets the same first treatment: swap in a better model. The answers barely move, because a RAG answer isn't produced by a model. It's produced by a pipeline with eight stations, and the model runs only one of them.
7/6/2026 · 7 min · jusCode · Read as Markdown
TL;DR
A RAG answer is produced by an eight-layer pipeline, and the model runs only one layer. Because the layers multiply, eight stations at ninety percent yield a forty-three percent system, which is why swapping models rarely helps.
- RAG is a pipeline with eight layers: ingestion, chunking, indexing, retrieval, reranking, context assembly, generation, evaluation.
- The Yield Chain: answer quality is the product of eight layer yields, not the score of one model. Eight stations at 90 percent each is a 43 percent system.
- Most quality is won or lost before the model sees a single token. The generation layer can't recover what retrieval never found.
- Debug in pipeline order, not model-first. When you measure per layer, the broken station is almost never number seven.
The ritual
The most expensive way to not fix a RAG system
The pattern repeats in every company. A team ships retrieval-augmented generation over their docs. The demo dazzles, production disappoints: answers are vague, sometimes wrong, occasionally invent things. So the team performs the ritual: upgrade to the newest model, rewrite the system prompt, raise the temperature, lower the temperature. Quality moves two points. Everyone concludes RAG is overrated.
Here's what actually happened. The right passage was chopped in half at chunking, so retrieval never surfaced it, so the model, which is layer seven of eight, answered from vibes. The team spent a quarter polishing the one station on the production line that was working fine. This post walks the whole line, station by station, and gives you the frame that makes the debugging obvious.
The stations
Layer by layer, with each one's favorite way to fail
Ingestion: what exists
Before anything can be retrieved it must be extracted, and extraction is where enterprise RAG quietly dies. PDFs with two-column layouts read in the wrong order. Tables flatten into word soup. Headers and footers repeat into every chunk. Whatever ingestion mangles, no downstream layer ever sees correctly, which makes this humble parsing step the highest-stakes station on the line.
yield killer: the revenue table your CFO asks about became one unreadable paragraph on day one.
Chunking: what's findable
Documents are split into retrievable units, and the split decides what questions can ever be answered. Chop at a fixed 500 tokens and answers straddle two chunks, so neither ranks. Respect semantic boundaries, headings, sections, paragraphs that complete a thought, and each chunk carries a whole idea. There is no universal chunk size; there are only boundaries that keep answers intact and boundaries that cut them in half.
yield killer: the answer exists in your corpus but was split across two chunks that each rank 40th.
Indexing: what's matchable
Chunks become searchable: dense embeddings for meaning, keyword indexes for exact terms, metadata for filtering. The classic mistake is going embeddings-only. Vector similarity is brilliant at "vacation policy" versus "time off rules" and blind at part numbers, error codes, and internal jargon your embedding model never saw. Index both, and keep it fresh: a stale index answers confidently from a world that no longer exists.
yield killer: a user searches for error E-4012 and semantic search cheerfully returns the philosophy of error handling.
Retrieval: what's seen
The search itself: given the query, fetch the top-k candidates, ideally from both indexes fused together. This is the ceiling-setter of the whole pipeline. If the right passage isn't in the candidate set, everything downstream is decoration, because no model can ground an answer in a passage it was never shown. The k is a knob worth respecting: too small and you miss, too large and you drown the next layers in noise.
yield killer: top-3 retrieval on a corpus where the right chunk reliably ranks fifth.
Reranking: what's first
First-stage retrieval optimizes for speed across millions of chunks; a reranker takes the shortlist and rereads each candidate against the query with full attention. It's the cheapest quality lever on the line, systematic evaluations of RAG configurations consistently find reranking among the highest-impact additions [2], and most teams skip it because their pipeline diagram came from a quickstart tutorial that skipped it too.
yield killer: trusting first-stage similarity scores, which were designed to be fast, to also be right.
Context Assembly: what's noticed
The survivors get ordered, deduplicated, and formatted into the prompt, and here hides a failure so sneaky it has its own research literature: models attend best to the beginning and end of long contexts and reliably miss what's buried in the middle [3]. Retrieve the perfect passage, place it as chunk 9 of 15, and you've hidden the answer from the model in plain sight. Position is not a detail. Position is signal.
yield killer: the right passage was retrieved, ranked, included, and ignored, because it sat in the dead zone.
Generation: how it's said
The model, at last. Its real job in RAG is narrower than people think: stay grounded in the provided context, cite what it used, and abstain when the context doesn't contain the answer. That last behavior is the difference between a system that says "I don't have that" and one that improvises. Notice how little of this is intelligence and how much is discipline, which is why model upgrades move RAG quality so much less than everyone expects.
yield killer: the model answers from its training data instead of your documents, and nobody can tell which happened.
Evaluation: what's true
The layer that makes the other seven fixable. Measure the final answer only and every failure looks like a model failure, which is exactly the trap from our opening story. Measure per station, did retrieval surface the right chunk, did it survive reranking, did the answer stay grounded in it, and the broken layer identifies itself. Evaluation isn't the last step of the pipeline. It's the feedback loop that runs the factory.
yield killer: a single end-to-end score, which tells you that something is broken and nothing about what.
Ingestion decides what exists. Retrieval decides what's seen. Generation decides how it's said. When answers are bad, walk the line in order and find the first station whose yield drops. In our experience and in the published measurements alike, it is almost never station seven.
The unique insight
The Yield Chain
Here's the frame that makes everything above actionable. Treat each layer as a production station with a yield: the fraction of the time it does its job. Ingestion parses cleanly 90 percent of the time. Chunking keeps answers intact 90 percent of the time. And so on, eight times. Because the layers are in series, the system's quality is roughly the product of the yields, and eight stations at an individually respectable 90 percent multiply out to a 43 percent pipeline. This single piece of arithmetic explains the entire lived experience of RAG: why demos work (easy questions pass every station), why production disappoints (real questions find the weak stations), and why the model-swap ritual fails (upgrading station seven from 90 to 95 percent moves the system from 43 to 45). The teams that win at RAG aren't the ones with the best model. They're the ones who know their eight numbers.
Zooming out
Where this sits in the bigger stack
In the 11 layers of the AI inference stack, this entire pipeline is a zoom into Layer 05, the context layer: the machinery that decides what the model actually sees. And it composes with everything we wrote about loops: an agent loop that retrieves on every iteration inherits all eight yields on every pass, which is one more reason loop budgets exist. Retrieval quality isn't a feature of your RAG product. It's load-bearing for everything you build above it.
References
The research behind the line
- Lewis et al., 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 20). The paper that named the pattern: generation grounded in retrieved evidence. arXiv:2005.11401
- Wang et al., 2024. Searching for Best Practices in Retrieval-Augmented Generation (EMNLP 24). The systematic study of RAG configurations, and the empirical case for hybrid retrieval and reranking. arXiv:2407.01219
- Liu et al., 2023. Lost in the Middle: How Language Models Use Long Contexts (TACL 24). The dead-zone result: models miss information buried mid-context, which makes assembly order a quality decision. arXiv:2307.03172
First in our RAG engineering series. Foundation: The 11 Layers of the AI Inference Stack. Related: our loop engineering series. Next: Silent Failures, how to evaluate a RAG system before your users do.
Test yourself
1. Eight layers each at 90 percent yield. Roughly what does the system deliver?
2. The answer exists in your docs but never appears in any answer. Where do you look first?
3. The right passage was retrieved and included, but the answer ignores it. Most likely cause?
FAQ
- Long context windows are huge now. Can't I just stuff everything in and skip RAG?
- For a small, static corpus, sometimes. At scale the arithmetic returns: stuffing a million tokens per query costs orders of magnitude more than retrieving five relevant chunks, and the mid-context dead zone gets worse as the haystack grows, not better. The mature pattern is both: retrieval to select, long context to give the selection room to breathe.
- Do I need a vector database on day one?
- No, and starting without one is often clarifying. A keyword index over well-chunked documents is an embarrassingly strong baseline, and building it first forces you through ingestion and chunking, the two layers that decide the most and get skipped the most. Add embeddings when semantic paraphrase becomes your measured failure mode, not before.
- How much does reranking actually help?
- Per dollar and per line of code, more than any other mid-pipeline change, which is why empirical best-practice studies keep landing on it [2]. It adds tens of milliseconds and typically converts a chunk that ranked seventh into the chunk that ranks first, which, given the dead-zone effect one layer later, is often the whole ballgame.
- What's the right chunk size?
- The honest answer: the one your evaluation layer says, for your corpus, this month. Legal contracts, API docs, and support tickets want different boundaries. The transferable rule isn't a number; it's that semantic boundaries beat fixed sizes, and that you should be able to defend your chunking with a measured retrieval-recall figure rather than a tutorial default.