loading…

Eval-Driven Development

Eval-Driven Development (EDD) is the generative AI analogue of Test-Driven Development (TDD). Coined in enterprise practice by airbnb and formalised in methodology by hamel husain and shreya shankar, EDD treats evaluation as an ongoing development loop rather than a post-hoc verification gate. [source: airbnb-eval-driven-development-2026]

Core Thesis: The One Rule

“When in doubt, look at your data.” — Rohit Girme et al. (Airbnb)

Product quality in agentic systems is primarily determined by error analysis over execution traces, not off-the-shelf benchmark metrics. Teams consistently report spending 60–80% of project effort on error discovery and annotation rather than automated scaffolding. [source: hamel-husain-ai-evals-faq-2026] Passing 100% of an eval suite indicates a benchmark that under-stresses system boundaries rather than proven reliability.

The Three-Layer Eval Funnel

flowchart TD
    subgraph Layer1 [Layer 1: Programmatic & Deterministic]
        P1[Strict Schema Validation / Types]
        P2[Forbidden Regex & Output Filters]
        P3[Deterministic Syntax / AST Checks]
    end

    subgraph Layer2 [Layer 2: Calibrated LLM-as-a-Judge]
        J1[3–5 Sharp Single-Dimension Judges]
        J2[Few-Shot Rubric & Binary Scoring]
        J3[Judge Calibration: TPR / TNR / Bias Checks]
    end

    subgraph Layer3 [Layer 3: Human-in-the-Loop]
        H1[Failure Trace Discovery]
        H2[Disagreement Adjudication]
        H3[High-Blast-Radius Gate Sign-off]
    end

    Input[Agent Execution Trace / Output] --> Layer1
    Layer1 -->|Pass / Fast Sub-second| Layer2
    Layer2 -->|Flagged Discrepancy or High Risk| Layer3
    Layer3 -->|New Failure Mode| Layer1

1. Programmatic & Deterministic Checks (Layer 1)

Fast, zero-LLM-cost filters that eliminate unviable generations before expensive evaluation:

• Strict JSON schema enforcement (zod, Pydantic) to prevent downstream parsing failures.
• Syntax, AST linting, length bounds, regex, and type-system checks.

2. Calibrated LLM-as-a-Judge (Layer 2)

Focused virtual judges targeting nuanced quality criteria (faithfulness, conciseness, instruction adherence):

• Single-Dimension Rule: Deploy 3–5 small, sharp evaluators evaluating one orthogonal property each, rather than one omnibus grader. [source: airbnb-eval-driven-development-2026]
• Separate Model Architecture: Always evaluate using a model distinct from or stronger than the generator.
• Statistical Calibration: Evaluate judges against expert ground-truth labels using true positive rate (TPR), true negative rate (TNR), and prompt-bias audits. [source: hamel-husain-shreya-shankar-evals-skills-2026]

3. Human Grounding & Adjudication (Layer 3)

Human attention is reserved for high-leverage boundaries:

• Inspecting sample traces (100 baseline runs) to categorise novel failure modes.
• Resolving edge-case disagreements and setting hard policy boundaries.

Five Operating Principles

1. Define Goals and Blocking Gates Upfront: Establish minimal pass thresholds before code generation.
2. Derive Metrics from Real Traces: Co-develop rubrics with domain stakeholders based on observed system failures.
3. Keep Evaluators Small and Sharp: Avoid monolithic prompt evaluators; isolate criteria.
4. Appoint an Accountable Human Decision-Maker: Explicitly designate an engineer to arbitrate ambiguous model behavior.
5. Continuous Calibration: Track judge drift whenever underlying foundation models or system prompts update.

Failure Modes

Symptom	Root cause	Fix
Eval suite passes but users complain	Benchmark under-stresses real boundaries	Add one eval case per real failure, forever
Evals written after the change	Post-hoc rationalization	Write the eval when writing the feature (TDD analogy)
Suite drifts from product reality	Prompts/features evolve without eval updates	Review eval fixtures each release cycle

Related

• eval taxonomy — the three-tier classification (deterministic, LLM-as-judge, human) underlying EDD.
• agentic code quality — multi-tier control architecture
• generator evaluator loop — iterative refinement loop
• error analysis and evals — diagnostic methodology
• hamel husain — evaluation authority
• shreya shankar — evaluator alignment research
• airbnb — industrial implementation