---
title: Cost, Performance, and Reliability
url: https://doc.liz6.com/en/ai/03-applications-and-production/03-cost-performance-and-reliability
locale: en
area: ai
tags:
- ai
- applications-and-production
date: 2026-06-30
modified: 2026-07-16
description: 'Integrating LLMs into production inevitably involves three things: billing (how to save on tokens), latency (how to avoid lag), and stability (how to handle rate limiting and errors). These topics were scattered across previous posts; this one consolidates them into an engineering checklist.'
---

# Cost, Performance, and Reliability

> Integrating LLMs into production inevitably involves three things: billing (how to save on tokens), latency (how to avoid lag), and stability (how to handle rate limiting and errors). These topics were scattered across previous posts; this one consolidates them into an engineering checklist.

## Overview

There is a gap between getting a prototype running and going live: engineering. In a demo, you just need to get a result from a single `messages.create` call; in production, you need to answer: How much will 10,000 calls cost? How many seconds will the user wait? What happens when you hit a 429? Will retries send an email twice?

The answers to these questions are scattered across previous posts: [prompt caching](/ai/01-models-and-context/01-context-engineering.md), [streaming and thinking](/ai/01-models-and-context/04-reasoning-and-thinking.md), [usage fields](/ai/02-Agent/05-evaluation-and-observability.md), and [choosing models by capability](/ai/01-models-and-context/03-model-architecture-dense-vs-moe.md). This post consolidates them along three lines: **cost / performance / reliability**, serving as a pre-launch checklist.

## Cost: Tokens Are the Only Billing Unit

Bill = input tokens × unit price + output tokens × unit price (output is usually 5× more expensive). The levers for saving money, ranked by return, are:

### 1. Prompt Caching — The Biggest Lever

In multi-turn conversations and agent loops, prefixes are highly repetitive. Cached parts are billed at **~0.1x** (see [Context Engineering](/ai/01-models-and-context/01-context-engineering.md)). Let's do the math (writing with a 5-minute TTL is 1.25x):

- No cache: 1.0x per call.
- Cache: 1.25x for the first write, then 0.1x for each read. **Break even in two calls** (1.25 + 0.1 = 1.35 < 2.0). With a 1-hour TTL, writing is 2x, requiring three calls to break even.

The prerequisite is that the prefix is **byte-for-byte stable**—do not insert `now()`/UUIDs into the system prompt, do not change the toolset mid-stream, and do not switch models, otherwise the cache fails silently. Verify using `usage.cache_read_input_tokens`; if it remains 0 for a long time, you have a "silent failure" culprit.

### 2. Choose the Right Model

Not all tasks require top-tier models. Use the [Models API](/ai/01-models-and-context/03-model-architecture-dense-vs-moe.md) to check capabilities and tier by cost: use cheaper models for simple classification/extraction, and Opus for complex reasoning/agents. In multi-agent scenarios, delegate subtasks to cheaper models (but **do not switch models within the same loop** as it destroys caching; start a separate subagent instead, see [Multi-Agent Orchestration](/ai/02-Agent/04-multi-agent-orchestration.md)).

### 3. Control Output and Thinking

Output is more expensive than input. Set `max_tokens` high enough to prevent truncation, but do not blindly max it out; use **effort** to adjust thinking depth (see [Reasoning and Thinking](/ai/01-models-and-context/04-reasoning-and-thinking.md)). Lower `effort` saves tokens—do not use reflective `xhigh` for simple tasks.

### 4. Batching — Half Price for Non-Real-Time Tasks

For jobs that do not require immediate return (offline analysis, batch classification, indexing), use the **Batches API**, where all tokens are **50% off**:

```python
batch = client.messages.batches.create(requests=[
    Request(custom_id="job-1", params=MessageCreateParamsNonStreaming(
        model="claude-opus-4-8", max_tokens=1024,
        messages=[{"role": "user", "content": "..."}])),
    # ... up to 100k requests / 256MB
])
# Poll batches.retrieve(id).processing_status until "ended" (usually within 1h, max 24h)
```

**Results return out of order—you must match them by `custom_id`. Never rely on order.** This is the same iron rule as "never attribute based on order" mentioned below.

### Estimate First

Before sending requests, you can estimate input volume using `count_tokens` (stateless, see [Tokens and Sampling](/ai/01-models-and-context/02-tokens-and-sampling.md)) and multiply by the unit price to predict costs; **do not use `tiktoken` to estimate Claude**, as it underestimates by 15–20%.

## Performance: Where Latency Comes From and How to Reduce It

Latency is primarily a function of output length (decoding each token one by one) + thinking overhead + network.

- **Streaming is the default.** For long inputs/outputs or large `max_tokens`, always use streaming: the first token starts returning immediately, improving perceived latency (TTFT) and **avoiding HTTP timeouts for non-streaming requests** (the SDK rejects non-streaming requests with large `max_tokens` because they might exceed 10 minutes and drop the connection). Use `.get_final_message()` / `.finalMessage()` for the complete result.
- **Effort and turn count are non-linear.** On agentic tasks, high effort often **reduces** the number of turns, making it faster and cheaper; do not assume "low effort is always faster." Sweep through levels on your own eval set (see [Evaluation and Observability](/ai/02-Agent/05-evaluation-and-observability.md)).
- **Caching is also a latency lever.** Cached prefixes skip recalculation. If the first request is slow, you can eliminate this with **warming** (send a `max_tokens: 0` request at startup to write the prefix into the cache).
- **Parallelism has cache timing pitfalls.** If N requests with the same prefix are sent **simultaneously**, none can read the cache others are still writing to, so all pay full price; send 1 first, wait for it to start streaming, then send the remaining N−1, so the latter can hit the cache.

## Reliability: Rate Limiting, Retries, and Idempotency

### Error Codes and Whether to Retry

| Code | Meaning | Retryable |
|---|---|:--:|
| 400 | Invalid request (params/format) | ❌ Fix request |
| 401 / 403 | Auth/Permission | ❌ |
| 404 | Wrong model ID or endpoint | ❌ |
| 429 | Rate limited (RPM/TPM/TPD exceeded) | ✅ Retry with backoff |
| 500 / 529 | Server error/overload | ✅ Retry with backoff |

Iron rule: **Do not retry 4xx (except 429)**—resending the same request will still fail. **Only retry 429/5xx with backoff.**

### Backoff and SDK Built-in Retries

429 responses include a `retry-after` header (seconds to wait). **The SDK already performs exponential backoff retries for 408/409/429/5xx by default (`max_retries=2`)**—in most cases, you don't need to write your own. Only customize if you need more aggressive/conservative behavior: exponential backoff + jitter, with a cap.

> Timeouts are also retried, so the worst-case wall-clock time can be `timeout × (max_retries+1)`. Factor this in when setting `timeout`.

### Idempotency: Retries Must Not Duplicate Side Effects

Backoff retries pose a risk with **side-effect tools** (sending emails, charging money, `git push`): if the first attempt actually succeeded but the response timed out, a retry sends two emails. The solution lies on the host side, not the model side:

- For dangerous/irreversible actions, use **human-in-the-loop** confirmation (see [Agent Loop](/ai/02-Agent/01-the-agent-loop-and-tool-use.md)).
- Side-effect operations should include an **idempotency key**: use the same key for the same logical action, and let the server deduplicate.
- Separate "deciding to do" from "actually doing": the model produces intent, and the host validates + executes idempotently.

### Attribution: Never Rely on Order for Concurrent/Batch Results

Concurrent requests and Batches results may return **out of order**. Use `custom_id` / trace id to map results back to requests (see [Evaluation and Observability](/ai/02-Agent/05-evaluation-and-observability.md)), **never rely on arrival order**—this is the most common bug causing mix-ups in batch scenarios.

## Best Practices

- **Enable prompt caching first, lock the prefix.** Put stable prefixes first, dynamic parts later; break even in two calls; monitor `cache_read_input_tokens` to verify hits.
- **Tier models by cost.** Use cheap models for simple tasks, Opus for complex ones; check capabilities via Models API, don't guess.
- **Use Batches for non-real-time tasks, half price.** Match results by `custom_id`, never rely on order.
- **Always stream long inputs/outputs.** Improves TTFT + avoids non-streaming HTTP timeouts; use `get_final_message` for complete results.
- **Only retry 429/5xx, not 4xx.** Prefer SDK built-in backoff, don't reinvent the wheel.
- **Make side-effect actions idempotent + require human confirmation.** Backoff retries × side effects = duplicate execution; use idempotency keys + human-in-the-loop as a safeguard.
- **Monitor cost/latency.** Put `usage` three fields, `stop_reason` distribution, and cache hit rate on dashboards (see [Evaluation and Observability](/ai/02-Agent/05-evaluation-and-observability.md)).

## Trade-offs and Failure Modes

- **Silent cache failure:** Timestamps in system prompt or mid-stream model switching cause `cache_read` to stay 0 for a long time, multiplying costs → Lock prefixes, debug via byte-for-byte diff.
- **Blindly using top-tier models:** Using Opus + `max` for simple tasks is expensive and slow → Tier by task, sweep effort.
- **Non-streaming with large `max_tokens`:** HTTP timeouts, connection dropped → Use streaming + `get_final_message`.
- **Retrying 4xx:** Resending the same request still fails, wasting quota → Only retry 429/5xx.
- **Retries triggering side effects:** Timeout retries send emails twice → Use idempotency keys + human confirmation for dangerous actions.
- **Attributing by order:** Batches/concurrency out-of-order causes mix-ups → Always use `custom_id`/trace id for matching.
- **Using `tiktoken` to estimate Claude costs:** Underestimates by 15–20% → Use `count_tokens`.

## References

- **Anthropic Official Docs:** Prompt Caching, Batch Processing, Rate Limits, Errors, Streaming, Models API, Token Counting (platform.claude.com, refer to official docs before implementation)
- **Related:** [Context Engineering](/ai/01-models-and-context/01-context-engineering.md), [Evaluation and Observability](/ai/02-Agent/05-evaluation-and-observability.md), [Agent Loop and Tools](/ai/02-Agent/01-the-agent-loop-and-tool-use.md), [Multi-Agent Orchestration](/ai/02-Agent/04-multi-agent-orchestration.md)

*Keywords: cost optimization, token billing, prompt caching economics, cache_read_input_tokens, Batches API, custom_id, out-of-order attribution, 50% discount, count_tokens, model selection, Models API, effort, streaming, TTFT, HTTP timeout, cache warming, max_tokens 0, rate limit, 429, retry-after, exponential backoff, exponential backoff, jitter, max_retries, idempotency, idempotency, human-in-the-loop, error codes, no retry for 4xx*