---
title: "Evals Skills"
section: "concepts"
type: "concept"
created: "2026-08-24"
updated: "2026-08-24"
confidence: "high"
tags: ["evaluation", "skills", "agents", "workflow"]
canonical: "https://pyweb.dev/wiki/evals-skills"
---
# Evals Skills

Eval methodology packaged as installable agent skills. Published by [hamel husain](/wiki/hamel-husain) and [shreya shankar](/wiki/shreya-shankar), **evals skills** are "a set of skills for AI product evals" that encode lessons "from helping 50+ companies and teaching students in our AI Evals course" as procedural instructions a coding agent loads on demand — [progressive disclosure](/wiki/progressive-disclosure) applied to evaluation practice.

The motivating critique: "Eval tools often get in the way. They nudge you toward generic off-the-shelf metrics and fully automated evals before you've looked at your data." Instead of a tool, the methodology ships as skills that steer the agent around "many easily avoidable footguns."

Scope note: these are product evals, "Not foundation model benchmarks like MMLU or HELM that measure general LLM capabilities. Product evals measure whether your pipeline works on your task with your data."

## The skill set

- **start** — entry point; "looks at your situation and routes you to the right skill"
- **eval-audit** — "Audit an eval pipeline and surface problems with prioritized severity"
- **error-discovery** — "Build a review app, select diverse samples, and organize your notes into failure modes"; builds a customized annotation interface and samples traces intelligently
- **generate-synthetic-data** — diverse synthetic test inputs "using dimension-based tuple generation"
- **write-judge-prompt** — design LLM-as-Judge evaluators for subjective quality criteria
- **validate-evaluator** — "Calibrate LLM judges against human labels using data splits, TPR/TNR, and bias correction"
- **evaluate-rag** — retrieval and generation quality in RAG pipelines
- **build-review-interface** — custom annotation interfaces for human trace review

Install: `npx skills add https://github.com/ai-evals-course/evals-skills`. The suggested prompt routes through `start` and, if `eval-audit` is picked, has the agent "investigate each diagnostic area using a separate subagent in parallel, then synthesize the findings into a single report" — a direct use of [subagents and context management](/wiki/subagents-and-context-management).

The authors frame the skills as "only a starting point. To make them better, tune them to be more specific to your data and domain" — consistent with the "look at your data" doctrine in [error analysis and evals](/wiki/error-analysis-and-evals).

## Related
- [error analysis and evals](/wiki/error-analysis-and-evals)
- [automated eval engineering](/wiki/automated-eval-engineering)
- [designing for verifiability](/wiki/designing-for-verifiability)
- [progressive disclosure](/wiki/progressive-disclosure)
- [subagents and context management](/wiki/subagents-and-context-management)

---

## Agent Navigation

cluster: person (170 pages) | betweenness: 128.4

### References (outbound)
- [Hamel Husain](https://pyweb.dev/wiki/hamel-husain.md)
- [Subagents and Context Management](https://pyweb.dev/wiki/subagents-and-context-management.md)
- [Error Analysis and Evals](https://pyweb.dev/wiki/error-analysis-and-evals.md)
- [Automated Eval Engineering](https://pyweb.dev/wiki/automated-eval-engineering.md)
- [Designing for Verifiability](https://pyweb.dev/wiki/designing-for-verifiability.md)
- [Progressive Disclosure](https://pyweb.dev/wiki/progressive-disclosure.md)

### Referenced by (inbound)
- [Agentic Engineering Patterns](https://pyweb.dev/wiki/agentic-engineering-patterns.md)
- [Skill Treatment Effect](https://pyweb.dev/wiki/skill-treatment-effect.md)
- [Structured Outputs](https://pyweb.dev/wiki/structured-outputs.md)

### Evidence (verified primary sources)
- [hamel-husain-evals-skills-for-coding-agents-2026](https://pyweb.dev/wiki/raw/articles/hamel-husain-evals-skills-for-coding-agents-2026.md) | origin: https://hamel.dev/blog/posts/evals-skills/ | ingested: 2026-08-24 | sha256: 0cf5ae6c8422ed833b8116575b539a129af0f9f715d5b584377b62cb9ab59cb9

### Machine endpoints
- Knowledge graph: https://pyweb.dev/api/graph.json
- Graph analysis: https://pyweb.dev/api/graph-analysis.json
- Context index: https://pyweb.dev/llms.txt