loading…

LLM Evals: Everything You Need to Know (AI Evals FAQ)

By Hamel Husain (with contributions from Shreya Shankar). Published at hamel.dev/blog/posts/evals-faq/ (2026).

Executive Summary & Core Philosophy

Evaluation is not a separate testing phase or an optional line item; it is the core iterative engine of AI product development. Unsuccessful AI products almost always suffer from a single root failure: lack of robust, domain-calibrated evaluation systems. In contrast to traditional software engineering where debugging is episodic, AI engineering requires continuous error analysis of raw model outputs and traces.

Named Frameworks & Patterns

1. TPR/TNR Calibration: Aligns LLM-as-a-judge against human ground truth on held-out datasets by optimizing True Positive Rate (sensitivity) and True Negative Rate (specificity) rather than aggregate accuracy or correlation coefficients.
2. The 100% Pass Rate Failure Mode: An evaluation suite showing a 100% pass rate is a diagnostic signal of an unchallenging, broken, or gamed evaluation. Meaningful evaluations maintain tension against the system boundary (e.g., target ~70% pass rates under stress).
3. Transition Failure Matrices: A diagnostic grid for multi-step agentic workflows where rows represent the last successful state and columns represent the first failure state (e.g., GenSQL -> ExecSQL vs DecideTool -> PlanCal), isolating error hotspots.
4. Benevolent Dictator Domain Expert: Concentrating quality arbitration in a single domain expert who understands user intent, avoiding multi-annotator committee noise and inter-annotator disagreement.
5. Critique Shadowing & Binary Pass/Fail: Requiring human and model judges to emit a binary pass/fail verdict paired with a concrete textual critique string, rather than 1-5 Likert scores.
6. Error Analysis Flywheel: The virtuous cycle linking trace inspection (Level 2), unit test assertions (Level 1), fine-tuning data curation, and targeted error remediation.

Core Decision Rules

• When to start evals: Spend 30 minutes manually reviewing 20–50 traces in a notebook before building automated infrastructure, because understanding empirical failure modes must precede metric selection.
• When choosing scoring scales: Use binary pass/fail judgments with textual critiques instead of 1–5 Likert scales, because Likert scales introduce extreme human variance and make LLM-as-a-judge calibration nearly impossible.
• When evaluating agentic workflows: Separate evaluation into two distinct tiers:
  1. End-to-end task success: Black-box evaluation of whether the user’s primary goal was achieved (exact answer, correct side-effect).
  2. Step-level diagnostics: Instrumented trace evaluation using transition failure matrices to pinpoint where intermediate state transitions broke.
• When selecting judge models: Start with the most capable frontier model to establish high TPR/TNR alignment against held-out human labels before attempting distillation or cost optimization.
• When evaluating RAG vs Agent systems: Evaluate retrieval and generation independently; never evaluate the generation step when retrieval has failed.

Anti-Patterns & Failure Mechanisms

• Generic Off-The-Shelf Metrics: Using out-of-the-box vendor metrics (ROUGE, BLEU, generic toxicity/helpfulness rubrics) that have zero calibration against actual product domain failure modes.
• Premature Automation: Building automated LLM judges or complex CI dashboards before conducting manual trace inspection.
• Committee Annotation Grids: Averaging scores from multiple non-expert annotators with vague rubrics, producing noisy labels that cannot train or calibrate judges.
• Pass-Rate Vanity: Designing evaluations to pass cleanly rather than actively probing failure boundaries.

Quantitative Claims & Qualifiers

• 60–80% of development time in production AI projects is spent on manual error analysis and trace inspection rather than model training or prompt tweaking.
• 700+ engineers and product managers trained across live AI Evals cohorts.
• ~70% pass rate is a healthier target for active development benchmarks than 100%, indicating genuine stress-testing of system edge cases.

What the Source Does NOT Claim

• Does not claim that LLM-as-a-judge can entirely replace human error analysis and domain oversight.
• Does not claim that foundation model benchmarks (MMLU, HELM, SWE-bench) predict product-level task reliability.
• Does not claim that 100% test pass rates represent high product quality.
• Does not claim that complex evaluation frameworks should precede simple notebook-based data inspection.