Evals Skills

Eval methodology packaged as installable agent skills. Published by hamel husain and shreya shankar, evals skills are “a set of skills for AI product evals” that encode lessons “from helping 50+ companies and teaching students in our AI Evals course” as procedural instructions a coding agent loads on demand — progressive disclosure applied to evaluation practice.

The motivating critique: “Eval tools often get in the way. They nudge you toward generic off-the-shelf metrics and fully automated evals before you’ve looked at your data.” Instead of a tool, the methodology ships as skills that steer the agent around “many easily avoidable footguns.”

Scope note: these are product evals, “Not foundation model benchmarks like MMLU or HELM that measure general LLM capabilities. Product evals measure whether your pipeline works on your task with your data.”

The skill set

start — entry point; “looks at your situation and routes you to the right skill”
eval-audit — “Audit an eval pipeline and surface problems with prioritized severity”
error-discovery — “Build a review app, select diverse samples, and organize your notes into failure modes”; builds a customized annotation interface and samples traces intelligently
generate-synthetic-data — diverse synthetic test inputs “using dimension-based tuple generation”
write-judge-prompt — design LLM-as-Judge evaluators for subjective quality criteria
validate-evaluator — “Calibrate LLM judges against human labels using data splits, TPR/TNR, and bias correction”
evaluate-rag — retrieval and generation quality in RAG pipelines
build-review-interface — custom annotation interfaces for human trace review

Install: npx skills add https://github.com/ai-evals-course/evals-skills. The suggested prompt routes through start and, if eval-audit is picked, has the agent “investigate each diagnostic area using a separate subagent in parallel, then synthesize the findings into a single report” — a direct use of subagents and context management.

The authors frame the skills as “only a starting point. To make them better, tune them to be more specific to your data and domain” — consistent with the “look at your data” doctrine in error analysis and evals.