Production Checklist

Tie all previous chapters into a single launch lifecycle: from "starting with an evaluation set" to "canary release," and then "adding a regression test every time something breaks in production." This is the final page of the AI section and serves as a handy checklist.

Overview

The previous dozen or so chapters covered models, context, agents, and application patterns in isolation. This chapter flips the script—weaving them together in the chronological order of launching an LLM/agent application. It serves both as a comprehensive wrap-up and as a tickable checklist.

Two core beliefs run throughout:

• No evaluation means no iteration. How do you know it "got better" after changing the prompt, swapping the model, or adding tools?—Rely on evals, not gut feeling (see Evaluation & Observability).
• Confirm whether you should launch before discussing how to launch. If you can hardcode it with a workflow, don't use an agent; if a single agent suffices, don't use multi-agent systems—the most robust system is always the simplest one.

Launch Lifecycle

flowchart TD
    A["① Choose Layer<br/>Workflow? Agent? Multi-agent?"] --> B["② Build Eval Set<br/>Small, realistic, includes edge cases"]
    B --> C["③ Tune prompt/model/effort<br/>Tune against the eval set"]
    C --> D["④ Install Safeguards<br/>Sandbox/Permissions/Injection/Secrets"]
    D --> E["⑤ Connect Observability<br/>Trace + Usage + Stop Reason"]
    E --> F["⑥ Canary Release<br/>Small traffic, rollback-capable"]
    F --> G["⑦ Production Failure<br/>Solidify into regression evals"]
    G --> C

Below are the checklist items for each stage, with each item pointing back to the chapter where it is detailed.

① Choose the Right Layer: Don't Rush to Agents

Before deploying an agent, pass through the four gates (if any answer is "no," step back to a simpler layer):

• Complexity: Is the task multi-step and impossible to fully specify in advance? (If you can hardcode the flow → use a workflow)
• Value: Is the result worth the higher cost and latency?
• Feasibility: Is the model actually good at this type of task?
• Error Cost: Can errors be detected and rolled back? (Testing/review/rollback mechanisms in place)

For multi-agent systems, add one more question: Only use multi-agent if a single agent + good tools can't solve it (see Multi-Agent Orchestration)—multi-agent introduces coordination overhead, context synchronization, and debugging complexity.

② Start with an Eval Set, Then Tune

The most important practice: Have a small, realistic eval set before you start. Even just 20 representative tasks (including known failures/edge cases, not just happy paths) is far superior to relying on a single gut feeling (see Evaluation & Observability).

• Eval set includes failure/edge cases, not just smooth scenarios.
• Scoring prioritizes reliability from high to low: Programmatic Validation > LLM-as-judge > Outcome scoring. If you can assert, don't hire a judge—Structured Output makes such validations stable and feasible.
• LLM judges are decoupled from the tested system (no self-grading), providing independently verifiable rubrics.

③ Tune Against Evals, Not Gut Feelings

• Prompt: Write "when/boundaries" rather than piling on "musts"; positive examples beat negative ones (see Prompt Engineering).
• Effort: Start with high, sweep medium/high/xhigh on the eval set to decide—the relationship is non-monotonic (see Reasoning & Thinking).
• Sampling Parameters: Removed for new models; delete temperature from old code and translate it into prompt + effort.
• Output Shape: If feeding downstream, use output_config.format / strict; don't rely on "please return JSON."
• Knowledge: Private/real-time knowledge goes through RAG; maximize recall@k first before tuning generation.
• Context Budget: Lock stable prefixes to secure caching, put dynamic content later, and use compaction/context editing for long conversations.

④ Install Safeguards: Model Outputs Are Untrusted

Before launch, install all the gates from Security & Safeguards:

• Bash/Tools: Isolated environment + whitelist + timeout + logging; commands are untrusted outputs.
• Paths: Normalize + validate boundaries, reject ../symlinks/URL-encoded traversal.
• Dangerous Actions: Human-in-the-loop for irreversible operations, graded by reversibility, least privilege.
• Injection: Treat tool outputs as data, not instructions; operator instructions go through role:system trusted channels.
• Secrets/PII: Never enter context/memory/prompts; credentials via proxy injection; multi-tenant isolation.
• Output Side: Check stop_reason before reading content (handle refusals); escape/parameterize before outputting to SQL/HTML/shell.

⑤ Connect Observability: Be Able to Answer "Where Did It Go Wrong?"

Launching blind is a disaster. Connect Evaluation & Observability:

• Trace/Span: One span per tool_use/tool_result/model call, enabling step-by-step attribution of latency and cost (OTel is initialized at site build time).
• Usage: Monitor input/output_tokens, cache_read_input_tokens (if consistently 0 = cache silently failing), cache_creation_input_tokens—total = sum of the three, don't look at just one field.
• Stop Reason Distribution: High max_tokens = output truncated; abnormal spike in refusal = prompt triggered refusal; many pause_turn = server-side tool loop continuing.
• Cost/Throttling: Cache hit rate, 429 frequency on dashboards (see Cost, Performance & Reliability).

⑥ Canary Release: Controlled, Rollback-Capable

• Streaming: Stream all long inputs/outputs to avoid non-streaming HTTP timeouts.
• Retry/Idempotency: Only retry 429/5xx; actions with side effects need idempotency keys + human confirmation.
• Attribution: Match concurrent/Batches results by custom_id/trace id, never by order.
• Small Traffic First: Canary new prompts/models on a small slice, compare eval vs. online metrics, then scale; keep rollback paths open.
• Watch Cache When Swapping Models: Swapping models invalidates the entire cache segment and requires re-baselining tokens; don't switch mid-loop in the main cycle (see Multi-Agent Orchestration).

⑦ Regression: Add a Defense Line Every Time Something Breaks

• Every failure found in production should be solidified into an eval case—this is how the eval set grows stronger over time (see Evaluation & Observability).
• Return to ③ to retune, run full evals to confirm no regression, then launch.

Best Practices (Condensed from the Whole Section)

• Eval first, iterate later. A small, realistic eval set is the judge for all changes; without it, all "optimizations" are gambling.
• Keep it simple. Workflow > Single Agent > Multi-Agent; each level up must justify its added coordination cost with corresponding benefits.
• Treat the window as a budget. Cache stable prefixes, retrieval over hard-packing, tune effort depth/shallowness—these three things control cost and quality.
• Model outputs are entirely untrusted. Security boundaries are on the host execution side: sandbox, path validation, dangerous action approval, secrets out of context.
• Quantify every step. Trace + usage + stop_reason make "what went wrong" visible; lacking observability means blind changes.
• Separate authoring and review lanes. The executing agent doesn't self-review; start a separate reviewer/verifier in an independent context (see Multi-Agent Orchestration).
• Production failure → Offline regression. Every incident becomes an eval; defense lines only increase, never decrease.

Trade-offs & Failure Modes

• Tuning without evals: Changing by gut feeling, unable to distinguish "got better" from "changed the wrong way" → Build 20 real evals first.
• Reflective agents/multi-agent: Coordination overhead > benefits, harder to tune → Pass the four gates, use workflow if possible.
• No observability on launch: Don't know what went wrong, burned how many tokens → Connect trace + usage + stop_reason first.
• Not monitoring cache hit rate: Costs quietly multiply before noticed → Add cache_read_input_tokens to the dashboard.
• Only testing happy paths: Crashes on edge cases in production → Eval set includes edge cases and grows with incidents.
• Executing agent self-evaluates: Self-endorsing → Review in a separate independent lane.
• Direct full-release of new prompts/models: No regression baseline, hard to rollback on issues → Canary + keep rollback path + run full evals.

References

• Anthropic Official Docs: Building Effective Agents, Building Evals, Observability, Handling Stop Reasons (platform.claude.com, refer to official docs before implementation)
• Full Section Links: Context Engineering, Tokens & Sampling, Model Architecture, Reasoning & Thinking, Agent Loop & Tools, MCP & Skills, Memory & State, Multi-Agent Orchestration, Evaluation & Observability, RAG & Retrieval Augmentation, Prompt Engineering & Structured Output, Cost, Performance & Reliability, Security & Safeguards

Keywords: Production, production checklist, launch lifecycle, eval-first, workflow vs agent, canary release, canary, rollback, regression, observability, trace, span, OTel, usage, stop_reason distribution, prompt caching, RAG, structured output, safeguards, sandbox, secret, human-in-the-loop, idempotency, authoring vs review lane, checklist