In almost every AI-training session I've run for teams putting retrieval-augmented generation into production, the conversation starts in the wrong place: which model should sit at the end of the pipeline (Claude, GPT, Gemini). By the end of the session, we're almost always talking about something upstream of the model entirely, because that's reliably where the actual failures were happening.
Where the failures actually live
A RAG system has, roughly, three places to fail: the retrieval step finds the wrong (or an incomplete) set of documents, the model reasons poorly over documents that were actually correct, or the output is right but phrased in a way nobody trusts. In every production system I've debugged, the overwhelming majority of visible failures trace back to the first one. Retrieval handed the model a plausible-looking but wrong or incomplete paragraph, and the model, doing exactly what it's supposed to do, reasoned confidently from bad input.
A great model reasoning over the wrong paragraph will produce a fluent, confident, wrong answer faster than a mediocre model would have. Model quality doesn't fix a retrieval problem. It makes a retrieval problem harder to notice.
What actually improves retrieval, in order of how often it mattered
- Chunking that respects the document's real structure, not a fixed token count that slices a clause in half. Most of the "the model hallucinated a policy detail" complaints I've debugged were actually "the policy detail was split across two chunks and only one got retrieved."
- Hybrid search, not pure semantic similarity. Exact terms (product names, account numbers, specific clause references) often need keyword matching that pure embedding similarity quietly deprioritizes in favor of "related-sounding" text.
- Re-ranking before the model ever sees the candidates, so the five documents that actually reach the prompt are the five most relevant, not the five that happened to embed closest to the query.
- Routing by query type, a lot like the model-routing I've written about for cost. A factual lookup and an open-ended reasoning question don't need the same retrieval depth, and treating them identically wastes both budget and accuracy.
In the workshops I run, teams consistently see a bigger accuracy jump from fixing chunking and adding a re-ranking step than from upgrading the model behind the pipeline, often by a wide margin and at a fraction of the cost.
The question I now ask before any model swap
"Have you actually looked at the ten documents retrieval handed the model on your last five wrong answers?" Most teams haven't, they've read the wrong final output and assumed the model reasoned poorly, without checking what it was actually given to reason over. Once you make that a habit, a surprising number of "model problems" turn out to be a retrieval system quietly handing a good model a bad hand, every time.