Cost and latency trade-offs

Each Claude call has three costs: money, wall-clock time, and rate-limit headroom. The levers are the same every time: which model, how much thinking, how many tokens in and out, and whether the answer is needed now. Evals (the previous lesson) tell you how far you can pull each lever before quality drops.

Model choice

The docs suggest two starting points:

• Efficiency-first: start with a fast, low-cost model (Claude Haiku 5.5) and move up only where evals show a capability gap. Good for high-volume, simple or latency-sensitive work such as log classification or alert dedup.
• Capability-first: start with the strongest practical model (Claude Opus 5.5) and optimise down later. Good for complex reasoning and long-running agentic work.

Need	Start with
Lowest latency and price, subagent tasks	claude-haiku-5-5
Everyday coding, analysis, agentic tool use	claude-sonnet-5-5
Complex agentic coding, long autonomy	claude-opus-5-5
The highest available capability	claude-fable-5-1

You can also combine models: an orchestrator on a strong model hands bulk work to cheaper workers, or a cheap executor escalates hard decisions to a stronger advisor.

Effort is often a better lever than switching models. Several models accept output_config.effort, which trades intelligence for latency and cost within one model. Thinking tokens are billed as output and are part of max_tokens, so lower effort on simple tasks saves real money.

Tokens in, tokens out

• Input: trim what you send (the last N log lines, grep for errors, retrieval instead of whole wikis) and cache the stable prefix.
• Output: max_tokens is a cap, not a target. It stops runaway answers, and it doesn’t count toward the output-tokens-per-minute rate limit, which only counts tokens actually generated. Ask for short formats (“five bullets”, “label only”) to bring the real output down.
• Tokenizer: Claude Opus 4.7 and later produce roughly 30% more tokens for the same text than earlier models. Recount when you migrate instead of reusing old budgets.

Batch what can wait

The Message Batches API processes requests asynchronously at 50% of the standard price. It suits nightly reports, eval runs, backfills and bulk classification:

• Most batches finish within an hour. Requests not processed within 24 hours expire, and results stay downloadable for 29 days.
• Each request needs a unique custom_id. Results can come back in any order, so always match by custom_id.
• Each request ends as succeeded, errored, canceled or expired. You aren’t billed for errored, canceled or expired requests.
• stream: true and fast mode don’t apply inside a batch.

# nightly_batch.py - summarise failed CI logs at batch pricing
import pathlib
import re
import time
import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY
logs = sorted(pathlib.Path("logs").glob("*.log"))

batch = client.messages.batches.create(requests=[
    {
        "custom_id": re.sub(r"[^A-Za-z0-9_-]", "-", p.stem)[:64],  # must match ^[a-zA-Z0-9_-]{1,64}$
        "params": {
            "model": "claude-haiku-5-5",
            "max_tokens": 400,
            "messages": [{"role": "user", "content":
                f"Summarise why this CI job failed in 3 bullets.\n<log>\n{p.read_text()[-20000:]}\n</log>"}],
        },
    }
    for p in logs
])

while (batch := client.messages.batches.retrieve(batch.id)).processing_status != "ended":
    time.sleep(60)

for entry in client.messages.batches.results(batch.id):
    if entry.result.type == "succeeded":
        text = "".join(b.text for b in entry.result.message.content if b.type == "text")
        print(f"== {entry.custom_id}\n{text}\n")
    else:
        print(f"== {entry.custom_id}: {entry.result.type}")

Latency: what the user waits for

• Streaming doesn’t make a request cheaper or the full answer faster. It sends the first tokens sooner, so an engineer watching a ChatOps reply sees progress immediately. It is also the recommended way to run long requests.
• Long requests: for anything that might run longer than about 10 minutes, or a large max_tokens, use streaming or the Batches API. Idle connections can be dropped by the network. The Python SDK refuses a non-streaming request it expects to exceed about 10 minutes.
• Smaller inputs and cache hits shorten time to first token. A smaller model or lower effort shortens generation time.

Recap

• Pick the smallest model and lowest effort that pass your evals. Combine models when most of the work is simple.
• max_tokens is a cap. Thinking is part of it and billed as output. Short output formats cut cost and latency.
• Batches cost 50% less, usually finish within an hour, expire at 24 hours, and return results in any order: match by custom_id.
• Streaming improves perceived latency, not cost. Use it, or batches, for long requests.