Prompt caching, and the traffic pattern where it costs you more

What is actually being cached

Before a model generates its first output token it must process every input token — the prefill phase. That work produces a large intermediate structure, the key-value cache, holding one key and one value vector per token per layer. For a 20,000-token prompt that is a substantial amount of computation and it is identical every time the prompt is identical.

Prompt caching stores that intermediate state on the provider's side and reuses it when a later request begins with the same tokens. You are not caching the response — same prompt, different sampling, still a different answer. You are caching the prefill work, which means you get both the cost reduction and the latency reduction, and the latency reduction is often the part users actually notice.

It is a prefix match, and that is the whole design

This is the one rule everything else follows from: the match starts at token zero and runs forward until the first difference. Everything after that difference is a miss. It is not a substring search, not a fuzzy match, and not per-block.

Request A: [system 4,000 tok][docs 16,000 tok][question 500 tok] Request B: [system 4,000 tok][docs 16,000 tok][different question] └──────── 20,000 tokens shared ────────┘ ← cache hit Request C: ["Today is 2026-07-31."][system][docs][question] └── one changed token at position 0 ──┘ ← 0% hit rate

Request C is the mistake that destroys more caching implementations than everything else combined. A timestamp, a request UUID, or a user's name interpolated into the top of a system prompt changes byte zero of the prefix on every single call, and nothing after it can ever be reused. The prompt looks 99.9% identical. The cache hit rate is exactly zero.

The corollary is an ordering rule you can apply mechanically: assemble the prompt in descending order of stability. Tool definitions and the frozen system prompt first, then static documents and few-shot examples, then conversation history, then the current user turn. Anything that varies per request goes last, always.

TTL: the window you have to hit

Cache entries expire. The standard window is five minutes from last use — each hit refreshes it — and providers typically also offer a longer one-hour option at a higher write cost. Five minutes is short enough that the traffic pattern of your application, not the size of your prompt, is what decides whether caching helps.

Two applications with identical prompts can land on opposite sides of this. A chat interface where a user sends a follow-up every thirty seconds hits the cache on every turn after the first. A weekly report generator that fires one request at a time, ten minutes apart, misses on every single one and pays the write premium each time for a cache nothing ever reads.

The cost arithmetic

Caching is not free to populate. Writing an entry costs more than a normal input token, and reading one costs dramatically less. The multipliers, relative to the base input price:

From those two multipliers you can derive the break-even directly. For N requests sharing one prefix within the TTL window, with W tokens in the shared prefix:

cached cost = 1.25·W + 0.1·W·(N−1) uncached cost = N·W set equal: 1.25 + 0.1(N−1) = N 1.15 = 0.9N N = 1.28 5-minute TTL: caching pays from the SECOND request onward. 1-hour TTL (2× write): break-even at N = 2.11 — you need a THIRD.

That is a low bar and it is why caching is close to free money in the common case. The trap is that N counts requests within one TTL window, not requests in total. Two hundred requests spread evenly across a day is N = 1 for every window.

A worked example, both directions

Take a support assistant: a 20,000-token stable prefix (system instructions, tool definitions, a product knowledge base) plus a 500-token variable user question. Two hundred requests in an hour.

BUSY HOUR — 200 requests, ~18 seconds apart, all inside the 5-min TTL No caching: 200 × 20,500 = 4,100,000 effective tokens With caching: 1 × 20,000 × 1.25 = 25,000 199 × 20,000 × 0.1 = 398,000 200 × 500 × 1.0 = 100,000 ───────── 523,000 effective tokens Reduction: 87.2%. At a $3-per-million input rate: $12.30 -> $1.57 per hour.

Now the same 200 requests, same prompt, spread evenly across a 24-hour day — one every 7.2 minutes, which is longer than the five-minute TTL. Every request finds an expired entry, writes a fresh one, and nothing ever reads it:

SPARSE DAY — 200 requests, 7.2 minutes apart, every one a cold write No caching: 200 × 20,500 = 4,100,000 effective tokens With caching: 200 × 20,000 × 1.25 = 5,000,000 200 × 500 × 1.0 = 100,000 ───────── 5,100,000 effective tokens 24% MORE EXPENSIVE than not caching at all.

Same prompt, same request count, same code — the only variable was arrival timing, and it moved the result from an 87% saving to a 24% penalty. This is why "turn on caching everywhere" is bad advice and why the hit rate is the only number that tells you which regime you are in.

When caching does nothing, or worse

• The prefix is below the minimum cacheable length. Providers enforce a floor, commonly in the 512 to 4,096 token range depending on the model, and it is not monotonic across model generations. Below it, the cache marker is silently ignored — no error, no entry, and a diagnostic field showing zero tokens written.
• Something varies at the front. A timestamp, a UUID, a session ID, a user's name in the system prompt. Byte zero differs, everything after it misses.
• Traffic is sparser than the TTL. The case above. Either batch requests together in time or switch to the longer TTL and accept the higher write cost.
• Tool definitions changed. Tools usually render at the very front of the prompt, so adding, removing, or reordering one invalidates everything. Serialise them deterministically — sort by name — because an unordered dict or set will reorder itself between runs and you will never see why.
• Non-deterministic JSON serialisation. json.dumps without sort_keys=True, or anything iterating a set, produces a byte-different prefix from identical data.
• You switched models. Caches are model-scoped. Routing between models mid-conversation means each one maintains its own entry and neither is warm.
• Concurrent identical requests. An entry only becomes readable once the first response begins streaming. Fire ten parallel requests with the same prefix and all ten pay the write. Send one, wait for its first token, then fan out the rest.
• You expected it to reduce output cost. It does not. Output tokens are billed at full price regardless, and on a short-prompt, long-answer workload caching is nearly pointless.

Measure it or you do not know

Every provider that offers caching reports it in the response usage object — typically a count of tokens written to cache, a count read from cache, and a count processed uncached. Log all three on every call and compute a hit rate. This is not optional instrumentation; a caching implementation you are not measuring is one you cannot distinguish from a broken one, and the broken version costs 25% more while looking identical in your code.

hit_rate = cache_read_tokens / (cache_read_tokens + cache_write_tokens + uncached_tokens) > 0.8 working as intended 0.3-0.8 partial — check whether traffic is sparser than the TTL ~ 0.0 something varies in the prefix, or it is below the minimum length # total prompt size = uncached + cache_creation + cache_read # an agent showing 4k "input tokens" after hours of work is not small — # the rest was served from cache. Sum the three, never read one.

If the hit rate is zero across repeated calls you believe are identical, do not theorise — diff the rendered prompts. Serialise the exact bytes sent on two consecutive requests to files and run a diff. The culprit is always visible and it is always something small and boring.

Architectural moves that raise the hit rate

• Freeze the system prompt. Never interpolate the date, the user, or a mode flag into it. Inject that context later in the message list, where invalidating from that point costs almost nothing.
• Batch by prefix. If you have a queue of jobs sharing a large prefix, sort the queue by prefix and run each group together. This is the single change that converts the sparse-traffic penalty into the busy-hour saving.
• Do not swap tools mid-conversation. Tools sit at position zero. If you need modes, pass the mode as message content rather than changing which tools exist.
• Make forked calls reuse the parent prefix exactly. Summarisation passes, sub-agents, and side computations often rebuild the system prompt from scratch and miss the cache they were sitting on. Copy the parent's prefix verbatim and append.
• Pre-warm only when the first request's latency is user-visible. A startup warm-up call is a real write charge; it is worth it for an interactive surface after a deploy and it is waste for a background job.
• Consider the longer TTL for bursty traffic. Gaps longer than five minutes but shorter than an hour are exactly the case the one-hour window exists for — at 2× write cost, so you need three hits per window instead of two.

The local equivalent

Local runtimes do the same thing under a different name. llama.cpp and Ollama keep the KV cache in memory between requests and reuse the matching prefix automatically, which is why a follow-up question in the same session starts generating almost immediately while a fresh session with the same context takes seconds. The mechanism is identical; the economics are not, because locally you are saving wall-clock time rather than money.

That makes the ordering discipline more important locally, not less, since prefill on Apple Silicon runs at a few hundred tokens per second and a 20,000-token prompt reprocessed from scratch is a genuinely long wait. Same rule, same reason: stable content first, volatile content last. Between that and retrieving fewer, better chunks — which is what a properly evaluated retrieval pipeline gives you — you can usually take a local RAG setup from unusable to pleasant without touching the model at all.

The short version

1. Caching stores prefill work, keyed on an exact prefix match from token zero.
2. Order the prompt stable-first: tools, system, static documents, history, then the current turn.
3. Writes cost about 1.25× base input on the short TTL; reads cost about 0.1×.
4. It pays from the second hit within the TTL window, and it costs you 24% more if you never get one.
5. Log the three usage counters and track a hit rate. Zero means something varies in your prefix.
6. It never reduces output cost. If your prompts are short and your answers long, this is not your lever.

Tools referenced in this guide

• 30B MoE on a MacBook — why prefix reuse matters even more locally, where prefill is the bottleneck.
• Embedding models for RAG — retrieving fewer chunks shrinks the prompt caching has to work on.
• Structured output without regret — the other half of making LLM calls cheap and reliable in production.
• Services — automation work built on exactly these cost and latency tradeoffs.