---
title: "Error Analysis and Evals"
description: "Qualitative inspection of production traces to discover failure modes, build domain-specific taxonomies, and derive trustworthy evaluation rubrics."
section: "concepts"
type: "concept"
created: "2026-08-22"
updated: "2026-08-30"
confidence: "high"
tags: ["evaluation", "workflow", "feedback-loops", "context-engineering"]
canonical: "https://pyweb.dev/wiki/error-analysis-and-evals"
---
# Error Analysis and Evals

Error analysis is the systematic process of reading, annotating, and categorizing real production failures before writing evaluation suites or prompt rubrics. Pioneered in traditional machine learning and adapted to modern AI systems by [hamel husain](/wiki/hamel-husain) and [shreya shankar](/wiki/shreya-shankar), it forms the foundational requirement for building defensible AI applications.


```text
1. collect failures   2. cluster by root cause
3. fix the biggest cluster   4. add eval case per failure
```

## The Error Analysis Lifecycle

1. **Dataset Collection:** Gathering representative production traces, multi-turn conversations, and tool execution logs.
2. **Open Coding:** Human domain experts review raw outputs and record unstructured notes on unexpected behaviors, UX friction, and first points of failure.
3. **Axial Coding:** Clustering open-ended observations into an actionable failure taxonomy and counting occurrence frequencies.
4. **Active Learning Integration:** Using AI coding skills to observe real-time human labeling, continuously update the failure taxonomy, and automatically retrieve high-probability candidate failure traces across large corpora.
5. **Targeted Metric Formulation:** Translating high-frequency taxonomy buckets into binary, task-specific evaluation criteria rather than generic 1–5 subjective Likert scores.

## Pitfalls of Skipping Error Analysis

- **The Vanity Metric Trap:** Tracking high-level generic scores (e.g. "helpfulness" or "coherence") that improve on paper while users continue to experience core workflow breakdowns.
- **Criteria Drift:** Writing rigid speculative rubrics before observing actual model outputs, leading to wasted labeling effort on non-existent failure modes.
- **Unvalidated LLM Judges:** Using uncalibrated models to score outputs without measuring judge precision, recall, and false positive rates against human ground truth.

## The Data Science Mapping (Husain, 2026)

In *The Revenge of the Data Scientist*, [hamel husain](/wiki/hamel-husain) argues every recurring eval pitfall is a missing data-science fundamental: reading traces and categorizing failures is Exploratory Data Analysis; validating an LLM judge against human labels is Model Evaluation; building representative test sets from production data is Experimental Design; getting domain experts to label outputs is Data Collection; monitoring production is Production ML. "The names changed, the work did not." [[source: hamel-husain-the-revenge-of-the-data-scientist-2026]](/wiki/raw/articles/hamel-husain-the-revenge-of-the-data-scientist-2026)

His five pitfalls: generic metrics, unverified judges (treat the judge like a classifier — human labels, train/dev/test partitions, precision/recall instead of accuracy), bad experimental design (ground synthetic test data in real logs; binary pass/fail over 1-5 Likert scales), bad data and labels (domain experts must label; "criteria drift" — validated by [shreya shankar](/wiki/shreya-shankar) and colleagues — means grading outputs is how criteria get defined), and automating too much (LLMs can wire up plumbing but cannot look at the data for you). The agent harness itself is data science: observability stacks of logs, metrics, and traces exposed to the agent are what let it tell when it is off track. [[source: hamel-husain-the-revenge-of-the-data-scientist-2026]](/wiki/raw/articles/hamel-husain-the-revenge-of-the-data-scientist-2026)

## Rule of Thumb

Read 100 failures before writing a single line of mitigation code -
the clusters, not the anecdotes, decide what to build.

## Related
- [automated eval engineering](/wiki/automated-eval-engineering)
- [closed loop agent improvement](/wiki/closed-loop-agent-improvement)
- [designing for verifiability](/wiki/designing-for-verifiability)
- [hamel husain](/wiki/hamel-husain)
- [shreya shankar](/wiki/shreya-shankar)
- [context engineering](/wiki/context-engineering)
- [agentic engineering patterns](/wiki/agentic-engineering-patterns)

---

## Agent Navigation

cluster: person (170 pages) | betweenness: 292.9

### References (outbound)
- [Hamel Husain](https://pyweb.dev/wiki/hamel-husain.md)
- [Shreya Shankar](https://pyweb.dev/wiki/shreya-shankar.md)
- [Automated Eval Engineering](https://pyweb.dev/wiki/automated-eval-engineering.md)
- [Closed-Loop Agent Improvement](https://pyweb.dev/wiki/closed-loop-agent-improvement.md)
- [Designing for Verifiability](https://pyweb.dev/wiki/designing-for-verifiability.md)
- [Context Engineering](https://pyweb.dev/wiki/context-engineering.md)
- [Agentic Engineering Patterns](https://pyweb.dev/wiki/agentic-engineering-patterns.md)

### Referenced by (inbound)
- [Automated Eval Engineering](https://pyweb.dev/wiki/automated-eval-engineering.md)
- [Closed-Loop Agent Improvement](https://pyweb.dev/wiki/closed-loop-agent-improvement.md)
- [Designing for Verifiability](https://pyweb.dev/wiki/designing-for-verifiability.md)
- [Eval-Driven Development](https://pyweb.dev/wiki/eval-driven-development.md)
- [Eval Taxonomy](https://pyweb.dev/wiki/eval-taxonomy.md)
- [Evals Skills](https://pyweb.dev/wiki/evals-skills.md)
- [Generator-Evaluator Loop](https://pyweb.dev/wiki/generator-evaluator-loop.md)
- [Airbnb](https://pyweb.dev/wiki/airbnb.md)
- [Epoch AI](https://pyweb.dev/wiki/epoch-ai.md)
- [Hamel Husain](https://pyweb.dev/wiki/hamel-husain.md)
- [Model Evaluation & Threat Research](https://pyweb.dev/wiki/model-evaluation-and-threat-research.md)
- [Shreya Shankar](https://pyweb.dev/wiki/shreya-shankar.md)
- [Tessl](https://pyweb.dev/wiki/tessl.md)
- [LLM Wiki Ecosystem: Spec vs. Implementations](https://pyweb.dev/wiki/llm-wiki-ecosystem-analysis.md)

### Evidence (verified primary sources)
- [parlance-labs-do-automated-evals-work-2026](https://pyweb.dev/wiki/raw/articles/parlance-labs-do-automated-evals-work-2026.md) | origin: https://parlance-labs.com/blog/posts/auto-evals/index.html | ingested: 2026-08-22 | sha256: 8a4ef31b67fcd59160d7ca6567554f676b70129841cb02787c805362547bcfcb
- [hamel-ai-product-engineering-notes-2026](https://pyweb.dev/wiki/raw/articles/hamel-ai-product-engineering-notes-2026.md) | origin: https://hamel.dev/notes/llm/ai-product-engineering/ | ingested: 2026-08-22 | sha256: 7b35f29cda741a46974fa2e5585b42d5e2e805566373b9e4a3d45ef42d10f2bb
- [hamel-husain-the-revenge-of-the-data-scientist-2026](https://pyweb.dev/wiki/raw/articles/hamel-husain-the-revenge-of-the-data-scientist-2026.md) | origin: https://hamel.dev/blog/posts/revenge/ | ingested: 2026-08-24 | sha256: a5c947dbab1261c1eb647d34f07e4b7e562438a2450d0178715070310a6ac8f5

### Machine endpoints
- Knowledge graph: https://pyweb.dev/api/graph.json
- Graph analysis: https://pyweb.dev/api/graph-analysis.json
- Context index: https://pyweb.dev/llms.txt