--- title: Evaluation and Observability url: https://doc.liz6.com/en/ai/02-Agent/05-evaluation-and-observability locale: en area: ai tags: - ai - Agent date: 2026-06-30 modified: 2026-07-16 description: 'Agents are hard to evaluate: outputs are open-ended, processes are non-deterministic, and workflows involve multiple steps. But without evaluation, iteration is impossible—evaluation (offline quality measurement) and observability (online runtime visibility) are complementary; only together can they transform an agent from "looks okay" to "measurable and improvable."' --- # Evaluation and Observability > Agents are hard to evaluate: outputs are open-ended, processes are non-deterministic, and workflows involve multiple steps. But without evaluation, iteration is impossible—evaluation (offline quality measurement) and observability (online runtime visibility) are complementary; only together can they transform an agent from "looks okay" to "measurable and improvable." ## Overview Agents are hard to evaluate for three reasons: outputs are **open-ended** (no single correct answer), processes are **non-deterministic** (different results for the same prompt), and workflows are **multi-step** (a single task may involve twenty tool calls). But **without evaluation, you cannot iterate**—when you change the prompt, switch models, or add tools, how do you know it "got better" rather than "broke differently"? This article covers two complementary practices: **evaluation (eval)**—offline, repeatable quality measurement; and **observability**—online visibility into what the agent is actually doing at runtime. The former answers "did it improve overall?", while the latter answers "which step went wrong?" Integrating both into your iteration loop allows agents to move from "looks okay" to "measurable and improvable." ## Evaluation: Offline Quality Measurement ```mermaid flowchart LR D["eval set
representative tasks + expectations/rubrics"] --> R["Run current version"] R --> J["Score"] J -->|Programmatic?| P["Code assertions"] J -->|Open-ended?| L["LLM-as-judge"] J -->|Has completion criteria?| O["Outcome scoring"] P --> I["Aggregate & Compare"] L --> I O --> I I -->|Modify prompt/model/tools| R ``` Use scoring methods in order of **reliability (highest to lowest)**: prefer the former over the latter whenever possible. ### 1. Programmatic Checks (Most Reliable) If it can be determined by code, never delegate it to a model. "Return JSON passing schema validation," "CSV has a numeric column `price`," "tests pass," "contains specific keyword fields"—use direct assertions. Deterministic, zero cost, no bias. Structured outputs (`output_config.format` / `strict: true`) make these checks stable and reliable (see [Prompt Engineering and Structured Outputs](/ai/03-applications-and-production/02-prompt-engineering-and-structured-output.md)). ### 2. LLM-as-Judge (Use Only for Open-Ended Tasks) For things like summary quality or relevance that cannot be judged programmatically, use one model as the judge. The key is **reducing bias**: - **Provide clear rubrics**: Break down "good or bad" into **independently verifiable criteria** ("did it cite sources?", "did it answer all three sub-questions?"), rather than letting the judge give a holistic score based on gut feeling. Vague rubrics → noisy scores. - **Separate judge from subject**: Don't let the same model grade its own output (self-endorsement). Use different models, or at least independent contexts, as judges. - **Beware of known biases**: Judges tend to favor longer answers, prefer their own model's style, and are influenced by option order—more specific rubrics mitigate these biases. - **Evaluate the scoring itself**: Sample and manually verify the judge's scores to confirm reliability before scaling up. ### 3. Outcome-Based (When Completion Criteria Exist) Managed Agents make "result-oriented evaluation" a primitive. Send a `user.define_outcome` event with: - `description` (task), `rubric` (`{type:"text"|"file"}`, **required**, write independently verifiable criteria), `max_iterations` (default 3, max 20). The platform runs an **independent grader** in an **iterate → grade → revise** loop, emitting `span.outcome_evaluation_start/ongoing/end` events each round. `end.result ∈ {satisfied, needs_revision, max_iterations_reached, failed, interrupted}`, continuing until criteria are met or the limit is reached. This builds "evaluation + automatic improvement" directly into the runtime. Suitable for tasks with clear "what completion looks like" (reports, models, documents). ### 4. Regression Every time you encounter a failure, **solidify it into an eval case**. This is how your eval set grows stronger over time—every online failure adds another offline defense line. **Most important practice**: **Start with a small, real eval set before tuning.** Even just 20 representative tasks (including known failures/edge cases, not just happy paths) is far better than guessing based on a single instance. ## Observability: Online Runtime Visibility Once an agent is live, you need to answer: What steps did it take? Which step was slow/failed? How many tokens were consumed? Why did it stop? - **Trace / Span**: Break a single run into visible steps—each `tool_use`/`tool_result`/thinking/model call is a span. Managed Agents provide event streams directly; `span.model_request_end` includes `model_usage` (token details for this inference), enabling step-by-step attribution of latency and cost. - **Usage (Cost + Cache Health)**: Each response's `usage` provides: | Field | What to look for | |------|--------| | `input_tokens` | Uncached input (full price) | | `output_tokens` | Output | | `cache_read_input_tokens` | Cache hits (~0.1x); **zero over time = silent invalidator** (see [Context Engineering](/ai/01-models-and-context/01-context-engineering.md)) | | `cache_creation_input_tokens` | Cache writes (~1.25x) | Note: `input_tokens` is only the uncached remainder; total = sum of all three—don't look at just one field and assume no tokens were spent. - **Stop Reason Distribution**: Proportions of `end_turn` / `tool_use` / `max_tokens` / `refusal` form a health dashboard. High `max_tokens` = output truncated (increase limit or use streaming); abnormal rise in `refusal` = prompt triggered refusal; many `pause_turn` = server-side tool loops continuing. - **OpenTelemetry**: Export traces/metrics to standard backends for unified viewing. > Real example: Our site's static generator initializes OTel during build—the log line `OTel tracer initialised, exporting to ...` confirms this. The same "trace every step, quantify every instance" approach applies directly to LLM applications. ## Integrating Both into the Loop build run (with trace) eval (score) Check trace to locate failure step modify Run again — Observability answers "which step failed", Evaluation answers "did it improve overall" Observability answers "**which step** went wrong"; evaluation answers "**overall** did it improve?" Without the former, you don't know why it failed; without the latter, you don't know if changes are real improvements. Missing either makes iteration blind guessing. ## Best Practices - **Start with a small, real eval set before tuning.** Even 20 representative tasks (including known failures/edge cases) is far better than guessing based on a single instance. - **Use scoring methods in order of reliability.** Programmatic checks > LLM-as-judge > outcome scoring; never delegate what can be `assert`ed to a model ([Structured Outputs](/ai/03-applications-and-production/02-prompt-engineering-and-structured-output.md) make these checks stable). - **Separate judge from subject, provide independently verifiable rubrics.** No self-endorsement; break down "good or bad" into itemized, verifiable criteria to reduce noise and bias. - **Solidify every failure into a regression eval.** Every online failure becomes an offline defense line, strengthening the eval set over time. - **Trace every step, quantify every instance.** Break runs into spans; attribute latency and cost step-by-step via `usage`/`stop_reason`; export to standard backends via OTel for unified viewing. - **Monitor cache hit rates.** `cache_read_input_tokens` consistently zero = silent invalidator, causing costs to quietly double. - **Look at aggregates for non-deterministic systems, don't be misled by single-instance feelings.** Pass rates, token distributions, `stop_reason` proportions; attribute batch results by `custom_id`/trace id, never by order. ## Trade-offs and Failure Modes - **Only testing happy paths**: Crashes on edge cases in production → eval sets must include failures/edge cases and grow with incidents (regression). - **Judge = Subject**: Self-endorsement → Separate judge from subject; corresponds to agent engineering's "**authoring and review in separate lanes**" (see [Multi-Agent Orchestration](/ai/02-Agent/04-multi-agent-orchestration.md)). - **Vague rubrics**: "Looks good" cannot be stably scored → Write **independently verifiable** itemized criteria. - **Looking at single instances instead of distributions**: Non-deterministic systems require aggregate views (pass rates, token distributions, stop_reason proportions); don't be misled by single-instance feelings. - **No attribution**: Batch/parallel results don't match specific instances → Use `custom_id` / trace id for attribution, never by order (Batches results are inherently unordered). - **Not monitoring cache hit rates**: Costs quietly multiply before discovery → Add `cache_read_input_tokens` to monitoring dashboards. ## References - **Anthropic Official Documentation**: Building Evals, Managed Agents Define Outcomes & Observability, Handling Stop Reasons (platform.claude.com) *Keywords: eval, eval set, programmatic check, structured outputs, LLM-as-judge, rubric, judge bias, outcome-based, user.define_outcome, grader, span.outcome_evaluation, max_iterations, regression, observability, trace, span, model_usage, usage, input_tokens, output_tokens, cache_read_input_tokens, stop_reason distribution, OpenTelemetry, OTel, authoring vs review lane, custom_id*