Airbnb

Airbnb is a global accommodation and travel platform whose engineering organisation has pioneered the methodology of eval driven development (EDD) for GenAI systems at scale. [source: airbnb-eval-driven-development-2026]

Contributions to GenAI Engineering

Airbnb established core industrial practices for evaluating generative AI systems, formalizing the Three-Layer Eval Funnel:

1. Layer 1: Fast, deterministic programmatic checks (schemas, AST, regex).
2. Layer 2: Small, sharp LLM-as-a-judge evaluators focused on single orthogonal dimensions.
3. Layer 3: High-leverage human evaluation for trace discovery and policy calibration.

Their foundational rule (“When in doubt, look at your data”) mandates inspecting 100 baseline execution traces to derive targeted evaluators rather than creating theoretical metrics in isolation.

Related

• eval driven development — core methodology
• agentic code quality — multi-tier quality architecture
• error analysis and evals — trace analysis
• hamel husain — evals methodology