Cost, Performance, and Reliability

Integrating LLMs into production inevitably involves three things: billing (how to save on tokens), latency (how to avoid lag), and stability (how to handle rate limiting and errors). These topics were scattered across previous posts; this one consolidates them into an engineering checklist.

Overview

There is a gap between getting a prototype running and going live: engineering. In a demo, you just need to get a result from a single messages.create call; in production, you need to answer: How much will 10,000 calls cost? How many seconds will the user wait? What happens when you hit a 429? Will retries send an email twice?

The answers to these questions are scattered across previous posts: prompt caching, streaming and thinking, usage fields, and choosing models by capability. This post consolidates them along three lines: cost / performance / reliability, serving as a pre-launch checklist.

Cost: Tokens Are the Only Billing Unit

Bill = input tokens × unit price + output tokens × unit price (output is usually 5× more expensive). The levers for saving money, ranked by return, are:

1. Prompt Caching — The Biggest Lever

In multi-turn conversations and agent loops, prefixes are highly repetitive. Cached parts are billed at ~0.1x (see Context Engineering). Let's do the math (writing with a 5-minute TTL is 1.25x):

• No cache: 1.0x per call.
• Cache: 1.25x for the first write, then 0.1x for each read. Break even in two calls (1.25 + 0.1 = 1.35 < 2.0). With a 1-hour TTL, writing is 2x, requiring three calls to break even.

The prerequisite is that the prefix is byte-for-byte stable—do not insert now()/UUIDs into the system prompt, do not change the toolset mid-stream, and do not switch models, otherwise the cache fails silently. Verify using usage.cache_read_input_tokens; if it remains 0 for a long time, you have a "silent failure" culprit.

2. Choose the Right Model

Not all tasks require top-tier models. Use the Models API to check capabilities and tier by cost: use cheaper models for simple classification/extraction, and Opus for complex reasoning/agents. In multi-agent scenarios, delegate subtasks to cheaper models (but do not switch models within the same loop as it destroys caching; start a separate subagent instead, see Multi-Agent Orchestration).

3. Control Output and Thinking

Output is more expensive than input. Set max_tokens high enough to prevent truncation, but do not blindly max it out; use effort to adjust thinking depth (see Reasoning and Thinking). Lower effort saves tokens—do not use reflective xhigh for simple tasks.

4. Batching — Half Price for Non-Real-Time Tasks

For jobs that do not require immediate return (offline analysis, batch classification, indexing), use the Batches API, where all tokens are 50% off:

batch = client.messages.batches.create(requests=[
    Request(custom_id="job-1", params=MessageCreateParamsNonStreaming(
        model="claude-opus-4-8", max_tokens=1024,
        messages=[{"role": "user", "content": "..."}])),
    # ... up to 100k requests / 256MB
])
# Poll batches.retrieve(id).processing_status until "ended" (usually within 1h, max 24h)

Results return out of order—you must match them by custom_id. Never rely on order. This is the same iron rule as "never attribute based on order" mentioned below.

Estimate First

Before sending requests, you can estimate input volume using count_tokens (stateless, see Tokens and Sampling) and multiply by the unit price to predict costs; do not use tiktoken to estimate Claude, as it underestimates by 15–20%.

Performance: Where Latency Comes From and How to Reduce It

Latency is primarily a function of output length (decoding each token one by one) + thinking overhead + network.

• Streaming is the default. For long inputs/outputs or large max_tokens, always use streaming: the first token starts returning immediately, improving perceived latency (TTFT) and avoiding HTTP timeouts for non-streaming requests (the SDK rejects non-streaming requests with large max_tokens because they might exceed 10 minutes and drop the connection). Use .get_final_message() / .finalMessage() for the complete result.
• Effort and turn count are non-linear. On agentic tasks, high effort often reduces the number of turns, making it faster and cheaper; do not assume "low effort is always faster." Sweep through levels on your own eval set (see Evaluation and Observability).
• Caching is also a latency lever. Cached prefixes skip recalculation. If the first request is slow, you can eliminate this with warming (send a max_tokens: 0 request at startup to write the prefix into the cache).
• Parallelism has cache timing pitfalls. If N requests with the same prefix are sent simultaneously, none can read the cache others are still writing to, so all pay full price; send 1 first, wait for it to start streaming, then send the remaining N−1, so the latter can hit the cache.

Reliability: Rate Limiting, Retries, and Idempotency

Error Codes and Whether to Retry

Iron rule: Do not retry 4xx (except 429)—resending the same request will still fail. Only retry 429/5xx with backoff.

Backoff and SDK Built-in Retries

429 responses include a retry-after header (seconds to wait). The SDK already performs exponential backoff retries for 408/409/429/5xx by default (max_retries=2)—in most cases, you don't need to write your own. Only customize if you need more aggressive/conservative behavior: exponential backoff + jitter, with a cap.

Timeouts are also retried, so the worst-case wall-clock time can be timeout × (max_retries+1). Factor this in when setting timeout.

Idempotency: Retries Must Not Duplicate Side Effects

Backoff retries pose a risk with side-effect tools (sending emails, charging money, git push): if the first attempt actually succeeded but the response timed out, a retry sends two emails. The solution lies on the host side, not the model side:

• For dangerous/irreversible actions, use human-in-the-loop confirmation (see Agent Loop).
• Side-effect operations should include an idempotency key: use the same key for the same logical action, and let the server deduplicate.
• Separate "deciding to do" from "actually doing": the model produces intent, and the host validates + executes idempotently.

Attribution: Never Rely on Order for Concurrent/Batch Results

Concurrent requests and Batches results may return out of order. Use custom_id / trace id to map results back to requests (see Evaluation and Observability), never rely on arrival order—this is the most common bug causing mix-ups in batch scenarios.

Best Practices

• Enable prompt caching first, lock the prefix. Put stable prefixes first, dynamic parts later; break even in two calls; monitor cache_read_input_tokens to verify hits.
• Tier models by cost. Use cheap models for simple tasks, Opus for complex ones; check capabilities via Models API, don't guess.
• Use Batches for non-real-time tasks, half price. Match results by custom_id, never rely on order.
• Always stream long inputs/outputs. Improves TTFT + avoids non-streaming HTTP timeouts; use get_final_message for complete results.
• Only retry 429/5xx, not 4xx. Prefer SDK built-in backoff, don't reinvent the wheel.
• Make side-effect actions idempotent + require human confirmation. Backoff retries × side effects = duplicate execution; use idempotency keys + human-in-the-loop as a safeguard.
• Monitor cost/latency. Put usage three fields, stop_reason distribution, and cache hit rate on dashboards (see Evaluation and Observability).

Trade-offs and Failure Modes

• Silent cache failure: Timestamps in system prompt or mid-stream model switching cause cache_read to stay 0 for a long time, multiplying costs → Lock prefixes, debug via byte-for-byte diff.
• Blindly using top-tier models: Using Opus + max for simple tasks is expensive and slow → Tier by task, sweep effort.
• Non-streaming with large max_tokens: HTTP timeouts, connection dropped → Use streaming + get_final_message.
• Retrying 4xx: Resending the same request still fails, wasting quota → Only retry 429/5xx.
• Retries triggering side effects: Timeout retries send emails twice → Use idempotency keys + human confirmation for dangerous actions.
• Attributing by order: Batches/concurrency out-of-order causes mix-ups → Always use custom_id/trace id for matching.
• Using tiktoken to estimate Claude costs: Underestimates by 15–20% → Use count_tokens.

References

• Anthropic Official Docs: Prompt Caching, Batch Processing, Rate Limits, Errors, Streaming, Models API, Token Counting (platform.claude.com, refer to official docs before implementation)
• Related: Context Engineering, Evaluation and Observability, Agent Loop and Tools, Multi-Agent Orchestration

Keywords: cost optimization, token billing, prompt caching economics, cache_read_input_tokens, Batches API, custom_id, out-of-order attribution, 50% discount, count_tokens, model selection, Models API, effort, streaming, TTFT, HTTP timeout, cache warming, max_tokens 0, rate limit, 429, retry-after, exponential backoff, exponential backoff, jitter, max_retries, idempotency, idempotency, human-in-the-loop, error codes, no retry for 4xx