Do Automated Evals Work?

Authors: Antaripa Saha, Hamel Husain URL: https://parlance-labs.com/blog/posts/auto-evals/index.html Date: July 11, 2026

Core Ideas

• Evaluated autonomous eval agents (Braintrust Loop, Arize AX Alyx, LangSmith chat agent, Claude Code, Codex, Factory Droid) against 100 human-annotated production traces from an apartment-leasing AI assistant.
• Top automated systems recovered up to ~87% of failures identified by domain experts, and uncovered real issues humans missed.
• Critical Limitations of Pure Automation:
  • Failed to catch interactions that looked technically correct but delivered broken user experience or subtle conversational failures.
  • Produced 10–20% false positives, flagging normal metadata or conversational pacing as defects.
  • Automated systems struggle because initial rubrics and definitions of “good” do not exist up front without examining raw data.
• Recommended Workflow: Human-in-the-Loop Active Learning:
  • Do not run error analysis as a one-shot batch.
  • Humans annotate traces iteratively; an AI tool (e.g. Shreya Shankar’s Error Discovery Skill) monitors annotations in real time, constructs a failure taxonomy, and proactively surfaces candidate traces matching emerging failure patterns.
  • “Look at your data”: domain expertise and error analysis are the primary differentiator in AI products.