Demystifying evals for AI agents

Authors: Mikaela Grace, Jeremy Hadfield, Rodrigo Olivares, Jiri De Jonghe (Anthropic Engineering)
Published: 2026-01-09
Source: Anthropic Engineering Blog
Contributors & Collaborators: Nicholas Carlini, David Hershey, Gian Segato, Mike Merrill, Alex Shaw, Ethan Dixon, Pedram Navid, Jake Eaton, Alyssa Baum, Lina Tawfik, Karen Zhou, Alexander Bricken, Sam Kennedy, Robert Ying; partners including Stripe, Cognition, Bolt, Sierra, Descript.

Executive Summary & Core Thesis

Good evaluations enable teams to ship AI agents with confidence. Without evals, agent development falls into reactive feedback loops: discovering bugs in production where fixing one regression silently introduces others.

Evaluating AI agents is fundamentally harder than evaluating single-turn models. Agents operate over many turns—calling tools, modifying external environment state, and adapting to intermediate outputs. In agent systems, mistakes compound non-linearly across turns. Furthermore, evaluating an agent means evaluating the agent harness (scaffold) and the model working together.

Formal Structural Model of Agent Evaluations

Task (Problem / Test Case): Single test unit with defined inputs, initial environment state, and success criteria.
Trial: Individual execution attempt on a task. Multiple trials are required to measure variance from non-deterministic generation and branching paths.
Grader: Scoring logic evaluating transcript or outcome assertions. Tasks typically employ multiple graders.
Transcript (Trace / Trajectory): Complete historical record of a trial, including all messages, tool calls, tool outputs, and reasoning steps.
Outcome: Final state of the environment at trial conclusion (e.g., verifying a database reservation record exists, rather than trusting an agent transcript asserting “flight booked”).
Evaluation Harness: Infrastructure that orchestrates tasks concurrently, provides mock environments, records trajectories, executes graders, and aggregates metrics.
Agent Harness (Scaffold): Runtime system enabling the model to act as an agent (e.g. Claude Code harness, Agent SDK primitives).
Evaluation Suite: Curated collection of tasks measuring targeted capabilities or behavioral guardrails.

Grader Taxonomy & Trade-offs

Grader Type	Methods	Strengths	Weaknesses
Code-based	String checks, binary tests (fail-to-pass / pass-to-pass), static analysis, outcome state verification, tool call parameter assertions	Fast, inexpensive, 100% deterministic, objective, easy to debug	Brittle to valid semantic variations; cannot evaluate open-ended or subjective qualities
Model-based	Rubric scoring, natural language assertions, pairwise comparison, reference comparison, multi-judge consensus	Highly flexible, scalable, captures nuance, handles freeform responses	Non-deterministic, token-expensive, susceptible to bias, requires human calibration
Human	Domain expert review, golden set annotation, disagreement adjudication	Gold standard for truth, handles high-stakes safety and edge nuance	Expensive, slow, unscalable for continuous CI/CD

Lifecycle Evolution of Agent Evals (Claude Code Case Study)

Stage 1 (Prototyping): Fast dogfooding, manual user feedback, and intuition.
Stage 2 (Narrow Targeted Evals): Automated evals for isolated, high-leverage failure modes (e.g. prompt concision, precise file editing diffs).
Stage 3 (Complex Behavioral Evals): Multi-turn behavioral evals for subtle failure modes (e.g. over-engineering, unnecessary refactoring, context compaction survival).
Stage 4 (Production & A/B Evals): Continuous telemetry, A/B testing, regression testing on static task banks.

Named Frameworks & Patterns (Verbatim)

“Scaffold-and-model joint evaluation thesis”: An evaluation scores the agent harness and model together; harness improvements alter benchmark scores as significantly as model weight upgrades.
“Outcome verification vs transcript assertion”: Verifying environmental ground truth (SQL state, git commits, passing test suites) rather than parsing the agent’s textual assertions.
“Multi-turn error compounding”: The compounding probability of failure as agents take multi-step branching tool actions.
“Creative loophole failure mode”: Frontier models discovering valid policy loopholes (e.g. Opus 4.5 solving tau2-bench booking tasks via unpredicted policy exceptions) that fail rigid static grading assertions.

Decision Rules

When evaluating an AI agent, do measure final environment outcomes (e.g., SQL records, file system state) rather than transcript assertions, because agents can assert success while failing to alter environment state.
When benchmarking agent performance, do execute multiple trials per task, because LLM non-determinism and multi-turn branching produce variance across runs.
When designing graders, do layer code-based deterministic checks first before using model-based rubrics, because code graders are fast, reproducible, and cheap while model graders add non-determinism.
When evolving agent harnesses (e.g. Claude Code), do start with narrow targeted evals (conciseness, file edits) before progressing to complex behavioral evals (over-engineering), because baseline harness stability must precede complex behavioral tuning.

Anti-Patterns & Failure Modes (with Mechanisms)

1. Transcript-Only Self-Reporting

Mechanism: Graders asserting success by parsing final LLM text output (“I fixed the bug”) rather than running environment tests or verifying modified code.
Observable Symptom: Agent hallucinates success, reports positive completion, but leaves broken code or empty database tables.

2. Single-Turn Benchmark Extrapolation

Mechanism: Assuming that benchmark scores on static, single-turn benchmarks (e.g. MMLU, standard coding puzzles) predict multi-turn agentic performance.
Observable Symptom: Models with high single-turn capability scores fail catastrophically in multi-turn environments due to lack of tool orchestration and state recovery capabilities.

What the Source Does NOT Claim

Does NOT claim that automated evaluations eliminate the necessity of human domain expert spot-checks.
Does NOT claim that single-turn benchmark metrics predict multi-turn agent task completion rates.