Prompt caching charges you less for input tokens you’ve sent before. It sounds automatic. It isn’t — the request has to be shaped right or it never hits.

How it actually works

When you send a request, the provider can cache the token prefix — the beginning portion of your prompt — and reuse that cached computation on a later request if the prefix matches exactly. “Exactly” is the operative word: even a single changed character upstream of the cache boundary invalidates it for everything after that point.

Cache hits typically cost a fraction of normal input token pricing — for both OpenAI and Anthropic, cached input tokens run around 90% cheaper than fresh ones, with a modest write cost the first time a prefix is cached. The saving only shows up on the second and later requests that reuse that prefix.

As two requests over time:

sequenceDiagram
    participant App
    participant API
    App->>API: Request 1 - static prefix plus question A
    API-->>App: Full price - prefix now cached
    App->>API: Request 2 - same static prefix plus question B
    API-->>App: Cache hit on prefix - about 90% cheaper
  

The request shape that makes it hit

Caching works on a shared, unchanging prefix. That means:

• Static content goes first. System prompt, tool definitions, retrieved reference documents, few-shot examples — anything that’s the same across requests belongs at the start of the message list.
• Variable content goes last. The user’s actual question, today’s date, session-specific data — anything that changes belongs at the end, after everything cacheable.
• Don’t interleave. If you put a timestamp or a random request ID inside your system prompt “for debugging,” you’ve broken caching for the entire prompt, because everything after that point in the prefix is now different every time.

A prompt structured as [static system prompt] + [static tool schema] + [static long document] + [changing user message] caches everything up to the user message. A prompt structured as [system prompt with today's date] + [document] + [user message] caches nothing, because the date invalidates the prefix on every single request.

Anthropic vs OpenAI, practically

Anthropic’s API requires you to explicitly mark cache breakpoints with cache_control on a content block — it’s opt-in per prompt. OpenAI’s caching is automatic for prompts over roughly 1024 tokens and requires no extra parameter, but you still only benefit if your prefix is actually stable across calls; automatic doesn’t mean guaranteed. Either way, the discipline is the same: structure static-before-variable and don’t put anything that changes above content you want cached.

# Anthropic — mark the reusable prefix explicitly
messages = [{
    "role": "user",
    "content": [
        {"type": "text", "text": long_reference_doc, "cache_control": {"type": "ephemeral"}},
        {"type": "text", "text": user_question},
    ]
}]

Where this pays off the most

• RAG with the same document set queried repeatedly — cache the retrieved context if the same chunks recur across users or sessions.
• Agent loops with a large, static tool schema sent on every turn — this is often the single biggest chunk of repeated input tokens in an agent, and it never changes turn to turn.
• Multi-turn chat — cache grows to cover the whole conversation history as it accumulates, so caching helps more as a conversation gets longer, not less.

The gotcha

Caches expire — typically after a few minutes of inactivity, sometimes longer for providers that offer an extended cache tier at a higher write cost. A low-traffic endpoint that gets one request every ten minutes will rarely see a cache hit no matter how well-structured the prompt is, because the cache has already cooled by the time the next request arrives. Caching earns you the most on high-frequency, shared-prefix traffic — check your actual request rate on an endpoint before assuming caching will move its bill.