Eval-driven development: Lessons from evaluating GenAI at scale

Authors: Rohit Girme, Dan Miller, Mia Zhao, Lifan Yang, Clint Kelly (Airbnb Tech Blog)
Published: 2026-07-28
Source: Airbnb Engineering
Cached Reference: /opt/data/cache/web/medium.com-1cf69c4430.md

Executive Summary & Core Thesis

Generative AI breaks standard software testing assumptions: LLM outputs are non-deterministic, correctness is subjective, evaluation often requires AI evaluating AI, and compound agent pipelines chain retrieval, reasoning, and tool calls that can fail independently.

Without a deliberate evaluation strategy, teams suffer false confidence (generic metrics look good while real failure modes slip to production), undetected regressions (prompt edits silently degrade unmeasured dimensions), and wasted engineering effort (scaling eval pipelines for metrics that do not correlate with product outcomes).

Airbnb formalizes Eval-Driven Development (EDD) as the GenAI analogue of Test-Driven Development (TDD): discovering, encoding, and continuously testing failure modes.

Foundational Principle: The One Rule

“When in doubt, look at your data.”
Manually review data and build intuition for success before building frameworks. Run prototypes through 100 examples (synthetic is fine), read the outputs and execution traces, categorize the model’s mistakes, and build targeted evals.

The 5 Principles of Eval-Driven Development (EDD)

Define goals and gates upfront: Establish optimization targets and mandatory pre-ship quality gates.
Let real errors guide your metrics: Co-develop metrics with cross-functional partners based on observed failures rather than inventing metrics in a vacuum.
Keep your evaluator set small and sharp: 3–5 well-calibrated LLM-as-judge evaluators beat 20–30 noisy ones. Each judge must evaluate exactly one correctness dimension.
Appoint a decision-maker: Appoint a designated human decision-maker to resolve disagreements on correctness rubrics.
Collaborate continuously: Maintain tight feedback loops where product partners regularly critique comparative outputs.

The 3-Layer Evaluation Funnel

Layer 1: Programmatic checks (fast, low resource — catches format/length/keyword failures)
                         ↓
Layer 2: LLM-as-a-Judge / Virtual Judges (nuanced — catches quality/faithfulness/tone issues)
                         ↓
Layer 3: Human evaluation (high resource — validates edge cases, calibrates the stack)

Layer 1 — Programmatic & Heuristic Metrics: Strict JSON schema validation, length bounds, forbidden keyword/regex scanning, classical classification metrics, and semantic similarity against gold references.
Layer 2 — Virtual Judges (LLM-as-a-Judge): Rubric-based scoring on single dimensions (faithfulness, tone, conciseness) using stronger models than the generator, with few-shot examples and chain-of-thought reasoning before scoring.
Layer 3 — Human Evaluation: 50–100 row golden datasets curated by domain experts, containing both positive and negative/bad examples to calibrate virtual judges.

Agentic System Evaluation: 3 Inspection Layers

Evaluating only the final output is insufficient for agentic systems because a correct final output can mask broken intermediate reasoning, invalid tool parameters, or inefficient trajectories.

Step-level: Evaluating individual tool calls and reasoning assertions.
Trajectory-level: Evaluating path efficiency, loop avoidance, and subagent invocation order.
Session-level: Evaluating whether the end-to-end user goal was achieved.

Implementation: Agents emit structured traces/spans under an application root; evaluation harnesses use depth-first search (DFS) tree traversal to reconstruct execution graphs in memory and evaluate subagent execution boundaries.

Quantitative Claims & Qualifiers

Exploratory sample size: 100 outputs and traces read manually during initial discovery.
Golden dataset size: 50–100 expert-labeled rows (must contain negative failure examples).
Virtual judge calibration threshold: Target high 80s to 90s% agreement (measured via Cohen’s kappa or Krippendorff’s alpha) against expert human annotations before deploying judges at scale.
Production monitoring sample: Daily continuous evaluation sampling 5% of live de-identified traffic.
Calibration case study: Initial virtual judge agreement of 78% was raised to 88% by refining rubric ambiguities and providing few-shot examples.

Decision Rules

When starting any GenAI or agent project, do manually inspect 100 outputs and traces, because discovering real failure modes is foundational to building meaningful metrics.
When designing LLM judges, do assign one distinct evaluation dimension per judge with clear scoring rubrics and separate models from the generator, because multi-criteria “God evaluators” produce noisy, un-actionable scores.
When calibrating automated judges, do include bad/failing examples in the 50–100 row golden set, because testing discrimination is impossible on positive examples alone.
When evaluating agentic systems, do evaluate across step-level, trajectory-level, and session-level via trace traversal, because correct final answers often mask broken intermediate reasoning or invalid tool arguments.
When human domain experts disagree on a rubric or label, do stop automation immediately and resolve human disagreement first, because uncalibrated or ambiguous rubrics produce compounding judge errors.

Anti-Patterns & Failure Modes (with Mechanisms)

1. Generic Metric False Confidence

Mechanism: Optimizing for off-the-shelf metrics (e.g. general “helpfulness” or BLEU/ROUGE) that fail to capture product-specific failure modes like hallucinated policy clauses.

2. Multi-Criteria “God Evaluator”

Mechanism: Writing a single massive judge prompt asking the LLM to score tone, correctness, conciseness, and formatting simultaneously.
Observable Symptom: High variance, poor correlation with human judgment, and inability to diagnose which specific dimension failed.

3. Positive-Only Golden Calibration Sets

Mechanism: Evaluating virtual judges only against known good outputs.
Observable Symptom: Inability to detect false positives (judges that rubber-stamp bad outputs), yielding inflated agreement metrics.

What the Source Does NOT Claim

Does NOT claim that virtual judges eliminate the need for human evaluation or that 100% human-judge agreement is attainable.
Does NOT claim that agent evaluation can be conducted purely as end-to-end black-box testing without inspecting intermediate tool trajectories.