Concept · Building
Prompt caching
The largest cost lever available to most people building with these models, and the one least likely to come up in a tutorial. It is also unusually easy to break by accident, in a way that shows up on the bill rather than in the output.
Last verified 11 August 2026 · No figures on this page; rates and lifetimes are vendor-specific and change, so check their docs
The problem it solves
A model has no memory between calls, so everything it needs must be sent every time. In any real application that means the same large preamble goes up over and over: your system instructions, your tool definitions, your style examples, the document being discussed, the conversation so far. → Context windows
You pay for all of it on every single call, and you wait for it to be processed on every single call. In a long conversation or an agent loop, the overwhelming majority of what you are paying for is material the model already saw a moment ago. → Tokens
Prompt caching lets the provider keep the processed form of that unchanging opening section and reuse it, so repeat sends are much cheaper and much faster. The savings on a workload with a big fixed preamble are not a rounding error, which is why it is worth understanding before you optimise anything else.
How it works, and the one rule that matters
Caching is prefix-based. It matches from the very start of your prompt forward, and Anthropic's documentation is explicit that a hit requires "100% identical prompt segments" up to the point you marked for caching. Not similar. Identical.
Everything else follows from that one property:
- Static content goes first, changing content goes last. The vendor guidance is to "place static content (tool definitions, system instructions, context, examples) at the beginning of your prompt." Anything that varies per request must sit after everything you want reused.
- A change invalidates everything after it. The prompt is treated in a fixed order, tools then system then messages, and "changes at each level invalidate that level and all subsequent levels." Edit a tool definition and the whole cache goes, not just the tool part.
- The reusable section has to actually be reusable. If a value in your opening section differs between requests, there is no shared prefix, and there is nothing to hit.
The mistake that quietly costs the most
Putting something that changes every request inside the section you meant to cache. The classic is a timestamp, a session ID or the user's message placed in the system preamble rather than after it. Every request then looks brand new, so the system writes a fresh cache entry and never reads one. Writing to cache is not free, so this is worse than not caching at all: you pay the premium every time and collect the benefit never.
It produces no error and no visible defect in the output. The only symptom is the bill, which is exactly why it survives so long in production.
Where the wins actually are
The pattern to look for is a large unchanging preamble followed by a small changing part, repeated many times.
- Agent loops. The best case there is. The system prompt and tool definitions get re-sent at every single step of the loop, unchanged, dozens of times per task. → Agents
- Long conversations. The history grows but its earlier portion never changes, so each turn can reuse everything up to the newest exchange.
- One big document, many questions. Put the document in the cached section and ask against it repeatedly.
- Retrieval systems, partly. The instructions and examples cache well. The retrieved passages change per question by definition, so they belong after the cached section, not inside it. → RAG and retrieval
And where it does nothing: one-off calls with short prompts, or workloads where every request genuinely differs from the start. There is no shared prefix to reuse, and no amount of configuration invents one.
What it is not
Three misunderstandings worth heading off, because all three lead somewhere expensive.
It is not memory. The model does not remember your conversation between sessions because it was cached. Caching stores the processed form of text you are still sending. Stop sending it and it is gone from the model's view entirely.
It does not free up context. Cached tokens still occupy the window exactly as they did before. This is a cost and latency optimisation, not a capacity one, and a prompt that overflows the window will still overflow it.
It does not persist for long, and it is not a database. Entries expire on a short timer measured in minutes by default, with longer options available at higher write cost, and the specifics differ by vendor and change over time. Anything that must survive reliably belongs in your own storage, not in a cache you do not control.
How to tell whether it's working
Do not assume, and do not infer it from the bill at the end of the month. Providers report per-call whether tokens were written to the cache or read from it, and that is the number to watch. A healthy repetitive workload shows one write followed by many reads. If you see writes on nearly every call and almost no reads, something in your supposedly-static section is changing, and finding it is usually the highest-value hour of work available to you.
The practical checklist
Put everything unchanging at the top, in a stable order. Keep anything per-request strictly after it. Mark the boundary at the end of the static material rather than on the varying part. Then check the reported read-versus-write counts on real traffic, not on a test call.
Sources
The prefix-matching behaviour, the requirement for "100% identical prompt segments", the instruction to place static content at the beginning, and the rule that "changes at each level invalidate that level and all subsequent levels" come from Anthropic's prompt caching documentation, fetched 11 August 2026. That page also documents cache lifetimes, minimum cacheable sizes and the pricing multipliers involved; those are deliberately not reproduced here because they are model-specific and change, and a stale copy of them would be worse than none.
Related → Agents · Tokens · Context windows · All → Concepts