Prompt caching isn't a magic flag you flip and forget. It's a technique with very specific mechanics -- and if you don't understand those mechanics, you'll implement it and keep paying full price without knowing why.
Prompt caching works on exact prefix matching. That means the system takes the exact bytes of the rendered prompt up to a cut point (`cache_control`) and compares them against the previous run. If a single byte changes anywhere in that prefix, everything after that point gets invalidated -- not just the part that changed.
Render order is always: tools (`tools`) → system prompt (`system`) → messages (`messages`). A breakpoint placed at the end of the system block caches tools + system together.
This has a huge practical implication: stable content must go *physically before* volatile content in your rendered prompt. If you put the current date or a UUID at the start of the system prompt, you invalidate the cache on every request no matter how many `cache_control` markers you add afterward.
For a one-hour TTL instead of five minutes: `{"type": "ephemeral", "ttl": "1h"}`. The max is 4 breakpoints per request, and they can be placed in any content block: system text, tool definitions, or blocks within messages.
A cache read costs roughly 10% of the normal input price. But a cache write isn't free: it costs 1.25x the normal price with a 5-minute TTL, and 2x with a 1-hour TTL. That means the break-even point depends on how many times you'll reuse that prefix:
- With a 5-minute TTL: you need at least 2 requests to come out ahead (1.25x + 0.10x = 1.35x, versus 2x uncached). - With a 1-hour TTL: you need at least 3 requests (2x + 0.20x = 2.2x, versus 3x uncached).
If your shared prefix is only used once, caching it costs you more, not less. The technique only pays off when there's real reuse -- multi-turn chat sessions, the same context document queried repeatedly, or the same tool set in an agent making many calls.
**System prompt shared across many requests.** A breakpoint at the last text block of the system prompt caches tools + system together -- the simplest pattern and the highest-impact one in most apps.
**Multi-turn conversations.** Put the breakpoint at the last content block of the most recent turn. Every subsequent request reuses the entire previous conversation prefix; earlier breakpoints remain valid read points, so hits accumulate incrementally as the conversation grows.
**Shared prefix with variable suffix** (few-shot examples + a different question each time, or retrieved context + user question): the breakpoint goes at the end of the *shared* part, not at the end of the full prompt.
When caching "isn't working" (`cache_read_input_tokens` at zero request after request), it's almost always one of these:
| Pattern | Why it breaks the cache | |---|---| | `datetime.now()` in the system prompt | The prefix changes on every request | | UUIDs or request IDs at the start of the content | Same problem -- every request is unique | | `json.dumps(d)` without `sort_keys=True`, or iterating a `set` | Non-deterministic serialization → the bytes differ even though the content is "the same" | | Conditional system sections (`if flag: system += ...`) | Each combination of flags is a different prefix | | A tool set that varies per user | Tools render at position 0 -- nothing caches across different users |
The way to diagnose this is simple: compare byte for byte the rendered prompt between two requests that "should" be identical. In 90% of cases the problem jumps out within the first 500 characters.
Not every change invalidates the entire cache. There are three levels: tools cache, system cache, and messages cache. Changing `tool_choice`, toggling `thinking` on/off, or adding images invalidates system and messages but not tools. Changing the model or a tool's definition does invalidate everything. Knowing this avoids the mistake of assuming any minor tweak forces you to rebuild the cache from scratch.
If first-request latency is visible to the user (interactive chat, not a batch job), you can eliminate the cost of that first cache-miss by sending a request with `max_tokens: 0` when your service starts up. The system runs the prefill -- writing the cache at your breakpoint -- and returns immediately with no output tokens billed.
It's only worth it if your traffic has gaps longer than the cache's TTL -- if real requests arrive less than 5 minutes apart, they already stay warm on their own, and additional pre-warming is pure extra spend.
Well implemented, prompt caching is the cost optimization with the highest return per hour of work in any production LLM architecture -- but only if you understand the prefix mechanics well enough not to break it without noticing.
Carlos Montiel is an enterprise AI solutions architect. He implements LLMs, Agents, RAG, and orchestrators for companies across Guatemala and Latin America. Reach out for a consultation.
Contact Carlos Montiel