Evals Skills for Coding Agents

By Hamel Husain & Shreya Shankar. Published at hamel.dev/blog/posts/evals-skills/ and github.com/ai-evals-course/evals-skills (2026).

Executive Summary & Purpose

Off-the-shelf evaluation tools frequently steer practitioners toward generic metrics and automated eval pipelines before they have inspected their actual data. evals-skills provides a structured, agent-executable procedural toolkit that automates sound evaluation hygiene: routing, error discovery, synthetic data generation, judge prompt authoring, and TPR/TNR calibration.

Named Skills & Architectural Components

start (Router): Entry point skill that inspects project state and routes the agent to the appropriate evaluation sub-skill based on available assets (raw traces vs existing pipelines).
eval-audit: Automated diagnostic auditor that inspects an existing evaluation pipeline for leakage, metric miscalibration, and unanchored scoring rubrics, producing prioritized severity reports.
error-discovery: Interactive tool that generates custom annotation interfaces, samples diverse edge-case traces, and clusters notes into systematic failure taxonomies.
generate-synthetic-data: Produces dimension-based tuple permutations to systematically stress-test multi-attribute agent prompts and tools without manual test authoring.
write-judge-prompt: Templates and structures domain-specific LLM-as-a-judge prompts around binary criteria, explicit counterexamples, and structured JSON critiques.
validate-evaluator: Executes calibration protocols comparing LLM-judge decisions against human-labeled validation splits, computing confusion matrices (TPR/TNR) and applying bias correction.
evaluate-rag: Evaluates retrieval precision, context sufficiency, and answer faithfulness in retrieval-augmented pipelines.
build-review-interface: Generates lightweight web/notebook annotation UIs customized to the project’s trace data schema.

Core Decision Rules

When traces exist without error categorization: Run error-discovery to inspect diverse sample slices and establish failure mode clusters before writing automated evaluators.
When an eval pipeline is already deployed: Run eval-audit across diagnostic areas using parallel subagents to identify rubric drift and metric contamination.
When authoring LLM judges: Use write-judge-prompt to enforce strict binary criteria with critique formatting, then execute validate-evaluator against a held-out human-annotated split.
When measuring judge performance: Measure True Positive Rate (TPR) and True Negative Rate (TNR) independently; reject judges that achieve high overall accuracy purely through majority-class imbalance.

Anti-Patterns & Failure Mechanisms

Uncalibrated Judge Deployment: Running LLM judges in CI/CD without held-out ground truth validation, leading to silent drift and false quality confidence.
Generic Metric Dependency: Using turnkey vendor scorecards rather than domain-calibrated evaluation skills.
Monolithic Agent Prompts: Attempting to instruct agents to perform evaluation without modular tool routing.

Quantitative Claims & Empirical Scope

Distills evaluation patterns and error distributions observed across 50+ enterprise companies and hundreds of cohort students.
Implements calibration protocols designed to prevent class imbalance illusions on evaluation benchmarks.

What the Source Does NOT Claim

Does not claim that automated skills eliminate the requirement for domain-expert human inspection during error discovery.
Does not claim that synthetic data alone suffices for production release without empirical trace verification.
Does not claim that judge prompts can be reused across disparate task domains without recalibration.