---
title: "Eval-Driven Development"
description: "Iterative, trace-grounded engineering discipline for discovering, encoding, and continuously testing GenAI failure modes."
section: "concepts"
type: "concept"
created: "2026-08-27"
updated: "2026-08-30"
confidence: "high"
tags: ["evaluation", "feedback-loops", "coding-guidelines", "workflow"]
canonical: "https://pyweb.dev/wiki/eval-driven-development"
---
# Eval-Driven Development

Eval-Driven Development (EDD) is the generative AI analogue of Test-Driven Development (TDD). Coined in enterprise practice by [airbnb](/wiki/airbnb) and formalised in methodology by [hamel husain](/wiki/hamel-husain) and [shreya shankar](/wiki/shreya-shankar), EDD treats evaluation as an ongoing development loop rather than a post-hoc verification gate. [[source: airbnb-eval-driven-development-2026]](/wiki/raw/articles/airbnb-eval-driven-development-2026)

## Core Thesis: The One Rule

> **"When in doubt, look at your data."** — Rohit Girme et al. (Airbnb)

Product quality in agentic systems is primarily determined by error analysis over execution traces, not off-the-shelf benchmark metrics. Teams consistently report spending **60–80% of project effort on error discovery and annotation** rather than automated scaffolding. [[source: hamel-husain-ai-evals-faq-2026]](/wiki/raw/articles/hamel-husain-ai-evals-faq-2026) Passing 100% of an eval suite indicates a benchmark that under-stresses system boundaries rather than proven reliability.

## The Three-Layer Eval Funnel

```mermaid
flowchart TD
    subgraph Layer1 [Layer 1: Programmatic & Deterministic]
        P1[Strict Schema Validation / Types]
        P2[Forbidden Regex & Output Filters]
        P3[Deterministic Syntax / AST Checks]
    end

    subgraph Layer2 [Layer 2: Calibrated LLM-as-a-Judge]
        J1[3–5 Sharp Single-Dimension Judges]
        J2[Few-Shot Rubric & Binary Scoring]
        J3[Judge Calibration: TPR / TNR / Bias Checks]
    end

    subgraph Layer3 [Layer 3: Human-in-the-Loop]
        H1[Failure Trace Discovery]
        H2[Disagreement Adjudication]
        H3[High-Blast-Radius Gate Sign-off]
    end

    Input[Agent Execution Trace / Output] --> Layer1
    Layer1 -->|Pass / Fast Sub-second| Layer2
    Layer2 -->|Flagged Discrepancy or High Risk| Layer3
    Layer3 -->|New Failure Mode| Layer1
```

### 1. Programmatic & Deterministic Checks (Layer 1)
Fast, zero-LLM-cost filters that eliminate unviable generations before expensive evaluation:
- Strict JSON schema enforcement (`zod`, Pydantic) to prevent downstream parsing failures.
- Syntax, AST linting, length bounds, regex, and type-system checks.

### 2. Calibrated LLM-as-a-Judge (Layer 2)
Focused virtual judges targeting nuanced quality criteria (faithfulness, conciseness, instruction adherence):
- **Single-Dimension Rule:** Deploy 3–5 small, sharp evaluators evaluating one orthogonal property each, rather than one omnibus grader. [[source: airbnb-eval-driven-development-2026]](/wiki/raw/articles/airbnb-eval-driven-development-2026)
- **Separate Model Architecture:** Always evaluate using a model distinct from or stronger than the generator.
- **Statistical Calibration:** Evaluate judges against expert ground-truth labels using true positive rate (TPR), true negative rate (TNR), and prompt-bias audits. [[source: hamel-husain-shreya-shankar-evals-skills-2026]](/wiki/raw/articles/hamel-husain-shreya-shankar-evals-skills-2026)

### 3. Human Grounding & Adjudication (Layer 3)
Human attention is reserved for high-leverage boundaries:
- Inspecting sample traces (100 baseline runs) to categorise novel failure modes.
- Resolving edge-case disagreements and setting hard policy boundaries.

## Five Operating Principles

1. **Define Goals and Blocking Gates Upfront:** Establish minimal pass thresholds before code generation.
2. **Derive Metrics from Real Traces:** Co-develop rubrics with domain stakeholders based on observed system failures.
3. **Keep Evaluators Small and Sharp:** Avoid monolithic prompt evaluators; isolate criteria.
4. **Appoint an Accountable Human Decision-Maker:** Explicitly designate an engineer to arbitrate ambiguous model behavior.
5. **Continuous Calibration:** Track judge drift whenever underlying foundation models or system prompts update.

## Failure Modes

| Symptom | Root cause | Fix |
|---|---|---|
| Eval suite passes but users complain | Benchmark under-stresses real boundaries | Add one eval case per real failure, forever |
| Evals written after the change | Post-hoc rationalization | Write the eval when writing the feature (TDD analogy) |
| Suite drifts from product reality | Prompts/features evolve without eval updates | Review eval fixtures each release cycle |

## Related
- [eval taxonomy](/wiki/eval-taxonomy) — the three-tier classification (deterministic, LLM-as-judge, human) underlying EDD.
- [agentic code quality](/wiki/agentic-code-quality) — multi-tier control architecture
- [generator evaluator loop](/wiki/generator-evaluator-loop) — iterative refinement loop
- [error analysis and evals](/wiki/error-analysis-and-evals) — diagnostic methodology
- [hamel husain](/wiki/hamel-husain) — evaluation authority
- [shreya shankar](/wiki/shreya-shankar) — evaluator alignment research
- [airbnb](/wiki/airbnb) — industrial implementation

---

## Agent Navigation

cluster: person (170 pages) | betweenness: 341.4

### References (outbound)
- [Airbnb](https://pyweb.dev/wiki/airbnb.md)
- [Eval Taxonomy](https://pyweb.dev/wiki/eval-taxonomy.md)
- [Agentic Code Quality](https://pyweb.dev/wiki/agentic-code-quality.md)
- [Generator-Evaluator Loop](https://pyweb.dev/wiki/generator-evaluator-loop.md)
- [Error Analysis and Evals](https://pyweb.dev/wiki/error-analysis-and-evals.md)
- [Hamel Husain](https://pyweb.dev/wiki/hamel-husain.md)
- [Shreya Shankar](https://pyweb.dev/wiki/shreya-shankar.md)

### Referenced by (inbound)
- [Agentic Code Quality](https://pyweb.dev/wiki/agentic-code-quality.md)
- [Red/Green TDD](https://pyweb.dev/wiki/red-green-tdd.md)
- [Site-as-Eval-Subject Grading](https://pyweb.dev/wiki/site-as-eval-subject-grading.md)
- [Airbnb](https://pyweb.dev/wiki/airbnb.md)
- [Agentic Code Quality — Evidence Map](https://pyweb.dev/wiki/agentic-quality-evidence.md)
- [Agentic Code Quality Cycle 2 — Entity Accounting](https://pyweb.dev/wiki/agentic-code-quality-cycle-2-entity-accounting.md)

### Evidence (verified primary sources)
- [airbnb-eval-driven-development-2026](https://pyweb.dev/wiki/raw/articles/airbnb-eval-driven-development-2026.md) | origin: https://medium.com/airbnb-engineering/eval-driven-development-lessons-from-evaluating-genai-at-scale-e817e5ae5788 | ingested: 2026-08-27 | sha256: 94414a24787cabca4537e9a34113139762e66eb17d3996269377298569a51b0c
- [hamel-husain-ai-evals-faq-2026](https://pyweb.dev/wiki/raw/articles/hamel-husain-ai-evals-faq-2026.md) | origin: https://hamel.dev/blog/posts/evals-faq/ | ingested: 2026-08-27 | sha256: fe8dd2419e8d96756f494a707be985227ae5724da945c9c4dcede5ca03f89154
- [hamel-husain-shreya-shankar-evals-skills-2026](https://pyweb.dev/wiki/raw/articles/hamel-husain-shreya-shankar-evals-skills-2026.md) | origin: https://hamel.dev/blog/posts/evals-skills/ | ingested: 2026-08-27 | sha256: c4fc84c125acf75690e6f51dcf104f5c29eff8e6619d2bd33e8f9aee3642008b
- [anthropic-demystifying-evals-for-ai-agents-2026](https://pyweb.dev/wiki/raw/articles/anthropic-demystifying-evals-for-ai-agents-2026.md) | origin: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents | ingested: 2026-08-27 | sha256: 0be1b99962fe19cbf994e5975425fcf6d7c9b898f31a75d883cb7a36d7db5ff9
- [openai-evaluation-best-practices-2026](https://pyweb.dev/wiki/raw/articles/openai-evaluation-best-practices-2026.md) | origin: https://developers.openai.com/api/docs/guides/evaluation-best-practices | ingested: 2026-08-27 | sha256: 5529f24f0c320d3e9e12bdb40a807d212a79ac922fb67b6447182c000206eed1

### Machine endpoints
- Knowledge graph: https://pyweb.dev/api/graph.json
- Graph analysis: https://pyweb.dev/api/graph-analysis.json
- Context index: https://pyweb.dev/llms.txt