RAG and retrieval

The problem it solves

A model knows what was in its training data and nothing else. It has never seen your company handbook, your support tickets, or anything published after its cutoff. Ask it about any of that and you get either an admission or an invention, and which one you get is not up to you. → Training cutoff

There is an obvious fix, and it is the one that works: put the relevant material in front of the model as part of the question. Everything a model can read in the moment, it can use. The only difficulty is that "the relevant material" might be ten thousand documents, and you cannot paste ten thousand documents into a question.

Retrieval-augmented generation is the name for solving that difficulty in the obvious way: before answering, go and find the few pieces that are actually relevant, then paste those in. Retrieve, then generate. That is the whole idea.

What actually happens

Ahead of time, your documents are cut into chunks, and each chunk is turned into a list of numbers that represents roughly what it is about. Those numbers are called an embedding, and their useful property is that passages about similar things end up with similar numbers, even when they share no words at all. "Cancelling your plan" and "how to end a subscription" land near each other.

Then, when a question arrives:

1. The question gets turned into numbers the same way.
2. The system finds the chunks whose numbers sit closest to it.
3. Those chunks get pasted into the prompt, above the question, usually with an instruction like "answer using only the material below."
4. The model reads all of it and writes an answer.

Step four is the part everyone photographs. Steps one to three are the part that determines whether the answer is any good.

Why most RAG failures are search failures

This is the single most useful thing on this page. If retrieval hands over the wrong chunks, no model can rescue the answer. It cannot know that the right passage exists somewhere it wasn't shown. It will do what it always does with insufficient material: produce something fluent and plausible from what it has. → Hallucination

So when a system built this way gives bad answers, the instinct is to blame the model, and the cause is usually one of these instead:

The chunking destroyed the meaning. The unglamorous decision that quietly sets your ceiling. Cut a table away from its heading and the numbers become anonymous. Cut a paragraph mid-argument and it retrieves as a fragment that says something the document never said. Anthropic's own engineering write-up puts the general problem plainly: splitting documents for retrieval "can lead to problems when individual chunks lack sufficient context," and their example is a chunk reporting that revenue grew by some percentage over the previous quarter, with nothing in it to say which company or which quarter. Retrieved on its own, it is worse than useless, because it is confidently about nothing.

Semantic search missed something exact. Embeddings are good at meaning and bad at literal strings. Part numbers, error codes, surnames, policy identifiers: these are precisely where "close in meaning" stops being the right question, and old-fashioned keyword matching does far better. This is why the standard advice is to run both and merge the results. Anthropic recommends the same combination, on the grounds that keyword search excels at exact matches while embeddings capture relationships, and the two are complementary rather than competing.

The answer was spread across many documents. Retrieval fetches passages that resemble the question. A question like "how has our refund policy changed over five years" has an answer that exists in no single passage, so nothing resembles it well enough to surface. Questions that require aggregating or counting across a whole corpus are a genuinely poor fit, and no amount of tuning changes that.

Nothing relevant existed, and the system answered anyway. Unless instructed otherwise, a model handed thin material will fall back on what it absorbed in training and blend the two, producing an answer that sounds sourced and isn't. The fix is boring and effective: instruct it to answer only from the supplied material, to say when the material doesn't cover the question, and to cite which passage each claim came from. Citations matter less as a courtesy to the reader than as a way to catch this failure at a glance.

"Can't I just paste everything in now?"

Increasingly, for small collections, yes, and it's the right first move. Windows have grown enough that a handbook or a contract set can simply be included wholesale. That removes the retrieval step, and with it every failure above. If your material fits, do that and skip the machinery. → Context windows

What makes it stop working is not a size limit so much as three costs that grow together. You pay for every token on every question, so re-sending an entire corpus each time is expensive in a way that scales with usage rather than with content. → Tokens Long inputs are slower. And a model asked to find one relevant line inside an enormous pile of mostly irrelevant material tends to do worse than one handed the few passages that matter, so more context is not automatically better answers.

The honest decision rule: paste it all in while you can, and reach for retrieval when the collection is too big, too expensive to re-send, or changing too often to keep pasting.

RAG is not fine-tuning, and the confusion is expensive

These get proposed interchangeably in meetings, and they solve different problems. Retrieval gives a model information at the moment it answers. Fine-tuning adjusts the model itself, and what it reliably changes is behaviour and form: tone, format, how to structure an output, how to follow a house style.

Trying to teach facts by fine-tuning is the expensive mistake in this neighbourhood. Facts trained in this way are diffuse rather than looked up, they cannot be cited, they cannot be corrected without retraining, and they go stale the moment the underlying document changes. When the requirement is "the assistant should know about our stuff," the answer is almost always retrieval. Fine-tuning has its own page, including why facts specifically fail to stick.

How to tell whether yours is working

The one practical habit worth adopting: test the search separately from the answer. Take a set of real questions, run only the retrieval step, and look at what came back. If the right passage isn't in there, nothing downstream can fix it, and you now know which half of the system to work on. Most teams skip this and spend weeks rewording prompts to compensate for a search problem.

After that, the highest-value improvements are usually the least sophisticated: chunk along the document's own structure rather than by character count, keep headings attached to what they head, add keyword search alongside the embeddings, and make the system show its sources so failures are visible instead of silent.

Sources

The chunk-context problem, the quoted phrasing about chunks lacking sufficient context, and the recommendation to combine keyword search with embeddings all come from Anthropic's Introducing Contextual Retrieval, fetched 11 August 2026. Its own proposed technique, prepending a short explanation to each chunk before indexing it, is a direct response to the first failure mode described above. The rest of this page is mechanical argument from how retrieval works rather than vendor claim.

Read first → Tool use · Related → Agents · Context windows · All → Concepts