---
title: Production Checklist
url: https://doc.liz6.com/en/ai/03-applications-and-production/05-production-checklist
locale: en
area: ai
tags:
- ai
- applications-and-production
date: 2026-06-30
modified: 2026-07-16
description: 'Tie all previous chapters into a single launch lifecycle: from "starting with an evaluation set" to "canary release," and then "adding a regression test every time something breaks in production." This is the final page of the AI section and serves as a handy checklist.'
---

# Production Checklist

> Tie all previous chapters into a single launch lifecycle: from "starting with an evaluation set" to "canary release," and then "adding a regression test every time something breaks in production." This is the final page of the AI section and serves as a handy checklist.

## Overview

The previous dozen or so chapters covered models, context, agents, and application patterns in isolation. This chapter flips the script—**weaving them together in the chronological order of launching an LLM/agent application**. It serves both as a comprehensive wrap-up and as a tickable checklist.

Two core beliefs run throughout:

- **No evaluation means no iteration.** How do you know it "got better" after changing the prompt, swapping the model, or adding tools?—Rely on evals, not gut feeling (see [Evaluation & Observability](/ai/02-Agent/05-evaluation-and-observability.md)).
- **Confirm whether you should launch before discussing how to launch.** If you can hardcode it with a [workflow](/ai/02-Agent/01-the-agent-loop-and-tool-use.md), don't use an agent; if a [single agent](/ai/02-Agent/04-multi-agent-orchestration.md) suffices, don't use multi-agent systems—the most robust system is always the simplest one.

## Launch Lifecycle

```mermaid
flowchart TD
    A["① Choose Layer<br/>Workflow? Agent? Multi-agent?"] --> B["② Build Eval Set<br/>Small, realistic, includes edge cases"]
    B --> C["③ Tune prompt/model/effort<br/>Tune against the eval set"]
    C --> D["④ Install Safeguards<br/>Sandbox/Permissions/Injection/Secrets"]
    D --> E["⑤ Connect Observability<br/>Trace + Usage + Stop Reason"]
    E --> F["⑥ Canary Release<br/>Small traffic, rollback-capable"]
    F --> G["⑦ Production Failure<br/>Solidify into regression evals"]
    G --> C
```

Below are the checklist items for each stage, with each item pointing back to the chapter where it is detailed.

## ① Choose the Right Layer: Don't Rush to Agents

Before deploying an agent, pass through the [four gates](/ai/02-Agent/01-the-agent-loop-and-tool-use.md) (if any answer is "no," step back to a simpler layer):

- [ ] **Complexity**: Is the task multi-step and impossible to fully specify in advance? (If you can hardcode the flow → use a workflow)
- [ ] **Value**: Is the result worth the higher cost and latency?
- [ ] **Feasibility**: Is the model actually good at this type of task?
- [ ] **Error Cost**: Can errors be detected and rolled back? (Testing/review/rollback mechanisms in place)

For multi-agent systems, add one more question: **Only use multi-agent if a single agent + good tools can't solve it** (see [Multi-Agent Orchestration](/ai/02-Agent/04-multi-agent-orchestration.md))—multi-agent introduces coordination overhead, context synchronization, and debugging complexity.

## ② Start with an Eval Set, Then Tune

**The most important practice: Have a small, realistic eval set before you start.** Even just 20 representative tasks (including known failures/edge cases, not just happy paths) is far superior to relying on a single gut feeling (see [Evaluation & Observability](/ai/02-Agent/05-evaluation-and-observability.md)).

- [ ] Eval set includes **failure/edge cases**, not just smooth scenarios.
- [ ] Scoring prioritizes reliability from high to low: **Programmatic Validation** > LLM-as-judge > Outcome scoring. If you can `assert`, don't hire a judge—[Structured Output](/ai/03-applications-and-production/02-prompt-engineering-and-structured-output.md) makes such validations stable and feasible.
- [ ] LLM judges are **decoupled from the tested system** (no self-grading), providing **independently verifiable** rubrics.

## ③ Tune Against Evals, Not Gut Feelings

- [ ] **Prompt**: Write "when/boundaries" rather than piling on "musts"; positive examples beat negative ones (see [Prompt Engineering](/ai/03-applications-and-production/02-prompt-engineering-and-structured-output.md)).
- [ ] **Effort**: Start with `high`, sweep `medium`/`high`/`xhigh` on the eval set to decide—the relationship is non-monotonic (see [Reasoning & Thinking](/ai/01-models-and-context/04-reasoning-and-thinking.md)).
- [ ] **Sampling Parameters**: Removed for new models; delete `temperature` from old code and translate it into prompt + effort.
- [ ] **Output Shape**: If feeding downstream, use `output_config.format` / `strict`; don't rely on "please return JSON."
- [ ] **Knowledge**: Private/real-time knowledge goes through [RAG](/ai/03-applications-and-production/01-rag-and-retrieval-augmented-generation.md); maximize recall@k first before tuning generation.
- [ ] **Context Budget**: Lock stable prefixes to secure [caching](/ai/01-models-and-context/01-context-engineering.md), put dynamic content later, and use compaction/context editing for long conversations.

## ④ Install Safeguards: Model Outputs Are Untrusted

Before launch, install all the gates from [Security & Safeguards](/ai/03-applications-and-production/04-security-and-protection.md):

- [ ] **Bash/Tools**: Isolated environment + whitelist + timeout + logging; commands are untrusted outputs.
- [ ] **Paths**: Normalize + validate boundaries, reject `..`/symlinks/URL-encoded traversal.
- [ ] **Dangerous Actions**: Human-in-the-loop for irreversible operations, graded by reversibility, least privilege.
- [ ] **Injection**: Treat tool outputs as data, not instructions; operator instructions go through `role:system` trusted channels.
- [ ] **Secrets/PII**: Never enter context/memory/prompts; credentials via proxy injection; multi-tenant isolation.
- [ ] **Output Side**: Check `stop_reason` before reading `content` (handle refusals); escape/parameterize before outputting to SQL/HTML/shell.

## ⑤ Connect Observability: Be Able to Answer "Where Did It Go Wrong?"

Launching blind is a disaster. Connect [Evaluation & Observability](/ai/02-Agent/05-evaluation-and-observability.md):

- [ ] **Trace/Span**: One span per `tool_use`/`tool_result`/model call, enabling step-by-step attribution of latency and cost (OTel is initialized at site build time).
- [ ] **Usage**: Monitor `input/output_tokens`, `cache_read_input_tokens` (if consistently 0 = cache silently failing), `cache_creation_input_tokens`—total = sum of the three, don't look at just one field.
- [ ] **Stop Reason Distribution**: High `max_tokens` = output truncated; abnormal spike in `refusal` = prompt triggered refusal; many `pause_turn` = server-side tool loop continuing.
- [ ] **Cost/Throttling**: Cache hit rate, 429 frequency on dashboards (see [Cost, Performance & Reliability](/ai/03-applications-and-production/03-cost-performance-and-reliability.md)).

## ⑥ Canary Release: Controlled, Rollback-Capable

- [ ] **Streaming**: Stream all long inputs/outputs to avoid non-streaming HTTP timeouts.
- [ ] **Retry/Idempotency**: Only retry 429/5xx; actions with side effects need idempotency keys + human confirmation.
- [ ] **Attribution**: Match concurrent/Batches results by `custom_id`/trace id, never by order.
- [ ] **Small Traffic First**: Canary new prompts/models on a small slice, compare eval vs. online metrics, then scale; keep rollback paths open.
- [ ] **Watch Cache When Swapping Models**: Swapping models invalidates the entire cache segment and requires re-baselining tokens; don't switch mid-loop in the main cycle (see [Multi-Agent Orchestration](/ai/02-Agent/04-multi-agent-orchestration.md)).

## ⑦ Regression: Add a Defense Line Every Time Something Breaks

- [ ] Every failure found in production should be **solidified into an eval case**—this is how the eval set grows stronger over time (see [Evaluation & Observability](/ai/02-Agent/05-evaluation-and-observability.md)).
- [ ] Return to ③ to retune, run full evals to confirm no regression, then launch.

## Best Practices (Condensed from the Whole Section)

- **Eval first, iterate later.** A small, realistic eval set is the judge for all changes; without it, all "optimizations" are gambling.
- **Keep it simple.** Workflow > Single Agent > Multi-Agent; each level up must justify its added coordination cost with corresponding benefits.
- **Treat the window as a budget.** Cache stable prefixes, retrieval over hard-packing, tune effort depth/shallowness—these three things control cost and quality.
- **Model outputs are entirely untrusted.** Security boundaries are on the host execution side: sandbox, path validation, dangerous action approval, secrets out of context.
- **Quantify every step.** Trace + usage + stop_reason make "what went wrong" visible; lacking observability means blind changes.
- **Separate authoring and review lanes.** The executing agent doesn't self-review; start a separate reviewer/verifier in an independent context (see [Multi-Agent Orchestration](/ai/02-Agent/04-multi-agent-orchestration.md)).
- **Production failure → Offline regression.** Every incident becomes an eval; defense lines only increase, never decrease.

## Trade-offs & Failure Modes

- **Tuning without evals**: Changing by gut feeling, unable to distinguish "got better" from "changed the wrong way" → Build 20 real evals first.
- **Reflective agents/multi-agent**: Coordination overhead > benefits, harder to tune → Pass the four gates, use workflow if possible.
- **No observability on launch**: Don't know what went wrong, burned how many tokens → Connect trace + usage + stop_reason first.
- **Not monitoring cache hit rate**: Costs quietly multiply before noticed → Add `cache_read_input_tokens` to the dashboard.
- **Only testing happy paths**: Crashes on edge cases in production → Eval set includes edge cases and grows with incidents.
- **Executing agent self-evaluates**: Self-endorsing → Review in a separate independent lane.
- **Direct full-release of new prompts/models**: No regression baseline, hard to rollback on issues → Canary + keep rollback path + run full evals.

## References

- **Anthropic Official Docs**: Building Effective Agents, Building Evals, Observability, Handling Stop Reasons (platform.claude.com, refer to official docs before implementation)
- **Full Section Links**: [Context Engineering](/ai/01-models-and-context/01-context-engineering.md), [Tokens & Sampling](/ai/01-models-and-context/02-tokens-and-sampling.md), [Model Architecture](/ai/01-models-and-context/03-model-architecture-dense-vs-moe.md), [Reasoning & Thinking](/ai/01-models-and-context/04-reasoning-and-thinking.md), [Agent Loop & Tools](/ai/02-Agent/01-the-agent-loop-and-tool-use.md), [MCP & Skills](/ai/02-Agent/02-mcp-and-skills.md), [Memory & State](/ai/02-Agent/03-memory-and-state.md), [Multi-Agent Orchestration](/ai/02-Agent/04-multi-agent-orchestration.md), [Evaluation & Observability](/ai/02-Agent/05-evaluation-and-observability.md), [RAG & Retrieval Augmentation](/ai/03-applications-and-production/01-rag-and-retrieval-augmented-generation.md), [Prompt Engineering & Structured Output](/ai/03-applications-and-production/02-prompt-engineering-and-structured-output.md), [Cost, Performance & Reliability](/ai/03-applications-and-production/03-cost-performance-and-reliability.md), [Security & Safeguards](/ai/03-applications-and-production/04-security-and-protection.md)

*Keywords: Production, production checklist, launch lifecycle, eval-first, workflow vs agent, canary release, canary, rollback, regression, observability, trace, span, OTel, usage, stop_reason distribution, prompt caching, RAG, structured output, safeguards, sandbox, secret, human-in-the-loop, idempotency, authoring vs review lane, checklist*