---
title: "Evals Skills for Coding Agents"
section: "raw"
type: "source"
created: "2026-08-27"
updated: "2026-08-27"
canonical: "https://pyweb.dev/wiki/raw/articles/hamel-husain-shreya-shankar-evals-skills-2026"
---
# Evals Skills for Coding Agents

By Hamel Husain & Shreya Shankar.
Published at hamel.dev/blog/posts/evals-skills/ and github.com/ai-evals-course/evals-skills (2026).

## Executive Summary & Purpose

Off-the-shelf evaluation tools frequently steer practitioners toward generic metrics and automated eval pipelines before they have inspected their actual data. `evals-skills` provides a structured, agent-executable procedural toolkit that automates sound evaluation hygiene: routing, error discovery, synthetic data generation, judge prompt authoring, and TPR/TNR calibration.

## Named Skills & Architectural Components

1. **`start` (Router):** Entry point skill that inspects project state and routes the agent to the appropriate evaluation sub-skill based on available assets (raw traces vs existing pipelines).
2. **`eval-audit`:** Automated diagnostic auditor that inspects an existing evaluation pipeline for leakage, metric miscalibration, and unanchored scoring rubrics, producing prioritized severity reports.
3. **`error-discovery`:** Interactive tool that generates custom annotation interfaces, samples diverse edge-case traces, and clusters notes into systematic failure taxonomies.
4. **`generate-synthetic-data`:** Produces dimension-based tuple permutations to systematically stress-test multi-attribute agent prompts and tools without manual test authoring.
5. **`write-judge-prompt`:** Templates and structures domain-specific LLM-as-a-judge prompts around binary criteria, explicit counterexamples, and structured JSON critiques.
6. **`validate-evaluator`:** Executes calibration protocols comparing LLM-judge decisions against human-labeled validation splits, computing confusion matrices (TPR/TNR) and applying bias correction.
7. **`evaluate-rag`:** Evaluates retrieval precision, context sufficiency, and answer faithfulness in retrieval-augmented pipelines.
8. **`build-review-interface`:** Generates lightweight web/notebook annotation UIs customized to the project's trace data schema.

## Core Decision Rules

- **When traces exist without error categorization:** Run `error-discovery` to inspect diverse sample slices and establish failure mode clusters before writing automated evaluators.
- **When an eval pipeline is already deployed:** Run `eval-audit` across diagnostic areas using parallel subagents to identify rubric drift and metric contamination.
- **When authoring LLM judges:** Use `write-judge-prompt` to enforce strict binary criteria with critique formatting, then execute `validate-evaluator` against a held-out human-annotated split.
- **When measuring judge performance:** Measure True Positive Rate (TPR) and True Negative Rate (TNR) independently; reject judges that achieve high overall accuracy purely through majority-class imbalance.

## Anti-Patterns & Failure Mechanisms

- **Uncalibrated Judge Deployment:** Running LLM judges in CI/CD without held-out ground truth validation, leading to silent drift and false quality confidence.
- **Generic Metric Dependency:** Using turnkey vendor scorecards rather than domain-calibrated evaluation skills.
- **Monolithic Agent Prompts:** Attempting to instruct agents to perform evaluation without modular tool routing.

## Quantitative Claims & Empirical Scope

- Distills evaluation patterns and error distributions observed across **50+ enterprise companies** and hundreds of cohort students.
- Implements calibration protocols designed to prevent class imbalance illusions on evaluation benchmarks.

## What the Source Does NOT Claim

- Does **not** claim that automated skills eliminate the requirement for domain-expert human inspection during error discovery.
- Does **not** claim that synthetic data alone suffices for production release without empirical trace verification.
- Does **not** claim that judge prompts can be reused across disparate task domains without recalibration.

---

## Agent Navigation

### Machine endpoints
- Knowledge graph: https://pyweb.dev/api/graph.json
- Graph analysis: https://pyweb.dev/api/graph-analysis.json
- Context index: https://pyweb.dev/llms.txt