Prompt caching

Many workloads send the same large block of text over and over: an on-call bot’s runbook, a long tool list, a policy document a reviewer checks every PR against. Prompt caching lets the API reuse the already processed prefix instead of reading it again. That cuts cost and time to first token. The output is the same either way.

The prefix is ordered

The cached prefix is built in a fixed order: tools → system → messages. A cache hit requires that everything up to the breakpoint is identical, byte for byte, to an earlier request.

This gives you a simple design rule: put stable content first and changing content last. Tool definitions and the runbook go at the front. The alert payload, the user’s question and anything with a timestamp go at the end.

Placing breakpoints

There are two ways to mark what to cache:

• Automatic caching: set a top-level cache_control: {"type": "ephemeral"}. The API puts the breakpoint on the last cacheable block and moves it forward as the conversation grows, which suits multi-turn chats.
• Explicit breakpoints: put cache_control on individual blocks in tools, system or messages. A request can have up to 4 breakpoints, and automatic caching uses one of them.

Put a breakpoint on the last block that stays the same across the requests that should share the cache, never on the incoming question itself.

# cached_runbook.py - cache a long runbook in the system prompt
import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY
runbook = open("runbook.md", encoding="utf-8").read()

def ask(question: str):
    resp = client.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=512,
        system=[
            {"type": "text", "text": "You are the on-call assistant for the payments platform."},
            {"type": "text", "text": runbook, "cache_control": {"type": "ephemeral"}},
        ],
        messages=[{"role": "user", "content": question}],
    )
    u = resp.usage
    total = u.cache_read_input_tokens + u.cache_creation_input_tokens + u.input_tokens
    print(f"write={u.cache_creation_input_tokens} read={u.cache_read_input_tokens} "
          f"uncached={u.input_tokens} total_input={total}")

ask("Checkout latency p99 is 4s. What do I check first?")
ask("The payments DB replica lag alert fired. Next step?")

What breaks a cache

Since the prefix is ordered, a change invalidates its own level and everything after it:

The usual mistakes are a timestamp or request ID at the top of the system prompt, tool definitions built from an unordered dictionary so their order changes, or a per-tenant line placed above the shared runbook. Each one makes every request a cache write and never a read.

Prompts below a minimum cacheable length (it depends on the model) are simply not cached, and no error is returned. If both cache fields are 0, check the length first.

Reading the usage fields

Total input is the sum of all three. A request with a 150k-token cached runbook and a 40-token question reports input_tokens: 40. Don’t mistake that for the size of the prompt. All three still count toward the context window. Caching changes what you pay for those tokens, not whether they take up space.

There is a throughput benefit as well. For most current models, cache reads don’t count toward the input-tokens-per-minute rate limit. Only uncached input and cache writes do. A high cache hit rate therefore increases your effective throughput as well as lowering cost.

In Claude Code, caching is automatic. /usage shows a Prompt cache line with the share of input served from cache and the misses. A miss right after compaction is expected, because the conversation was rewritten.

Recap

• Caching reuses an identical prefix built in the order tools → system → messages. Put stable content first and changing content last.
• Use automatic caching or up to 4 explicit cache_control breakpoints, placed on the last block that stays the same.
• A change at one level invalidates that level and every later one. Prefixes that are too short are silently not cached.
• Total input = cache_read_input_tokens + cache_creation_input_tokens + input_tokens. Cache reads usually don’t count toward ITPM.