Reliability patterns

An LLM call is a network dependency like any other, with the difference that the work behind it is slow and expensive. Treat it the way you would treat a flaky upstream service: classify errors, retry only what is transient, set timeouts, plan a fallback, and log enough to debug at 3 a.m.

Know your status codes

Code	Type	What to do
400	invalid_request_error	Fix the request. Also returned when a spend limit you set is hit
401	authentication_error	Fix the key. Retrying won’t help
403	permission_error	Fix access or workspace settings
404	not_found_error	Check the path, model ID or resource ID
413	request_too_large	Shrink the request
429	rate_limit_error	Wait for retry-after, then retry
500	api_error	Retry with exponential backoff
504	timeout_error	Retry; for long work use streaming or batches
529	overloaded_error	The API is overloaded across users. Back off and retry

Every response carries a request-id header, which the Python SDK exposes as ._request_id. Log it on every call and include it in support tickets.

Two kinds of 429

A 429 usually means you exceeded RPM, input tokens per minute or output tokens per minute, and retry-after says how long to wait. Limits use a token bucket: capacity refills continuously, and short bursts can trip a per-minute limit. A sharp jump in traffic can also trigger acceleration limits, so ramp traffic up gradually.

A 429 caused by reaching your tier’s monthly spend cap has no retry-after and keeps failing until access resumes. error.details.error_code is enforced_spend_limit_reached. Retrying is pointless here. Alert a human.

The anthropic-ratelimit-*-remaining and -reset response headers let you slow down before you get a 429.

Retries, backoff and timeouts

The official SDKs already retry connection errors, 408, 409, 429 and 5xx twice by default with exponential backoff, honouring retry-after. Know that default, and change it deliberately:

• max_retries: raise it for batch-style jobs. Set it to 0 if you run your own retry loop, so the two don’t multiply.
• timeout: the Python SDK default is 10 minutes. For an interactive bot, set something much shorter, and stream long generations instead of waiting on one blocking call.
• Add jitter to your own backoff so that a fleet of workers doesn’t retry in lockstep.

When streaming, an error can arrive after the HTTP 200, as an error event in the stream. Handle that path too.

Fallbacks and graceful degradation

When the retry budget runs out on 529s or timeouts, choose a deliberate outcome:

• Fall back to another model for a degraded but useful answer, for example a short triage summary instead of a full analysis. Your evals should tell you which fallback is good enough.
• Defer the work to a queue or the Batches API if it isn’t urgent.
• Fail loudly with an actionable message (“Claude unavailable; runbook link attached”) rather than posting an empty or half-finished reply.

Also check the stop reason, because an HTTP 200 doesn’t mean the answer is complete. max_tokens and model_context_window_exceeded mean the output was cut off. Don’t parse it as a finished JSON plan.

Idempotency

Retries repeat work. That is harmless for reading or summarising, and dangerous when a tool call has side effects. If the agent’s restart_service tool runs, then the next API call times out and is retried, the model may ask to restart again. Protect yourself in your own tools:

• Give each action an idempotency key (incident ID + action + target) and skip duplicates.
• Prefer declarative operations (“scale to 3”) over relative ones (“add 1 replica”).
• Record completed actions in state that survives retries and restarts.

Observability

# resilient_call.py - classify errors, back off with jitter, fall back, log structurally
import json, logging, random, time
import anthropic

logging.basicConfig(level=logging.INFO, format="%(message)s")
log = logging.getLogger("oncall-bot")
client = anthropic.Anthropic(max_retries=0, timeout=30.0)  # we own the retry loop
PRIMARY, FALLBACK = "claude-sonnet-5-5", "claude-haiku-5-5"

def is_transient(e: Exception) -> bool:
    if isinstance(e, anthropic.APIConnectionError):  # includes APITimeoutError
        return True
    return isinstance(e, anthropic.APIStatusError) and (
        e.status_code == 429 or e.status_code >= 500)  # 500, 504, 529 ...

def call(model: str, prompt: str, attempts: int = 4):
    for attempt in range(attempts):
        start = time.monotonic()
        try:
            r = client.messages.create(model=model, max_tokens=800,
                                       messages=[{"role": "user", "content": prompt}])
            log.info(json.dumps({"model": model, "request_id": r._request_id,
                                 "stop_reason": r.stop_reason, "in": r.usage.input_tokens,
                                 "out": r.usage.output_tokens,
                                 "ms": round((time.monotonic() - start) * 1000)}))
            return r
        except (anthropic.APIConnectionError, anthropic.APIStatusError) as e:
            if not is_transient(e):
                raise  # 400/401/403/404/413: fix the request, don't retry
            resp = getattr(e, "response", None)
            hint = resp.headers.get("retry-after") if resp is not None else None
            if isinstance(e, anthropic.RateLimitError) and hint is None:
                raise  # e.g. monthly spend cap: retrying will not help
            delay = float(hint) if hint else min(30, 2 ** attempt) * random.uniform(0.5, 1.5)
            log.warning(json.dumps({"model": model, "error": type(e).__name__,
                                    "attempt": attempt + 1, "sleep_s": round(delay, 1)}))
            time.sleep(delay)
    return None

prompt = "Summarise this alert for the on-call channel: disk 97% on db-replica-2"
resp = call(PRIMARY, prompt) or call(FALLBACK, prompt, attempts=2)
if resp is None:
    raise SystemExit("Claude unavailable after retries and fallback; page a human.")
print(next(b.text for b in resp.content if b.type == "text"))

Log per call: model, request ID, stop reason, token usage (including the cache fields), latency and retry count. Never log prompts that contain secrets or customer data. Alert on rising 429/529 rates, a falling cache hit rate, and a growing share of truncated answers.

Recap

• 4xx request errors need a fix, not a retry. 429, 500, 504, 529 and connection errors are transient.
• Honour retry-after. A spend-cap 429 has none and won’t clear by retrying.
• SDKs retry twice by default. Set max_retries and timeout deliberately and add jitter to your own loops.
• Plan fallbacks, check stop_reason, make side-effecting tools idempotent, and log request IDs and usage on every call.