---
title: "Do Automated Evals Work?"
section: "raw"
type: "source"
created: "2026-08-22"
updated: "2026-08-22"
canonical: "https://pyweb.dev/wiki/raw/articles/parlance-labs-do-automated-evals-work-2026"
---
# Do Automated Evals Work?

**Authors:** Antaripa Saha, Hamel Husain
**URL:** https://parlance-labs.com/blog/posts/auto-evals/index.html
**Date:** July 11, 2026

## Core Ideas

- Evaluated autonomous eval agents (Braintrust Loop, Arize AX Alyx, LangSmith chat agent, Claude Code, Codex, Factory Droid) against 100 human-annotated production traces from an apartment-leasing AI assistant.
- Top automated systems recovered up to ~87% of failures identified by domain experts, and uncovered real issues humans missed.
- **Critical Limitations of Pure Automation:**
  - Failed to catch interactions that looked technically correct but delivered broken user experience or subtle conversational failures.
  - Produced 10–20% false positives, flagging normal metadata or conversational pacing as defects.
  - Automated systems struggle because initial rubrics and definitions of "good" do not exist up front without examining raw data.
- **Recommended Workflow: Human-in-the-Loop Active Learning:**
  - Do not run error analysis as a one-shot batch.
  - Humans annotate traces iteratively; an AI tool (e.g. Shreya Shankar's Error Discovery Skill) monitors annotations in real time, constructs a failure taxonomy, and proactively surfaces candidate traces matching emerging failure patterns.
  - "Look at your data": domain expertise and error analysis are the primary differentiator in AI products.

---

## Agent Navigation

### Machine endpoints
- Knowledge graph: https://pyweb.dev/api/graph.json
- Graph analysis: https://pyweb.dev/api/graph-analysis.json
- Context index: https://pyweb.dev/llms.txt