Eval Taxonomy

LLM systems are probabilistic: no change is small, and whether the butterfly “flaps” or “Flaps” its wings may change the output. Traditional testing assumes deterministic input-to-output mapping; evals are the AI engineer’s unit tests — the tool for wrangling predictability from a probabilistic system and an indispensable part of productionizing any AI app.^[raw/aihero/what-are-evals.md]

Why Manual QA Fails Here

A demo is easy; production is not. Manual “try a few favorite prompts and see if it feels better” is dangerous when any change (model swap, prompt tweak) can affect the entire system. You need automated evaluation on every change to know whether you’re getting better or worse — the escape from the “Vibes-Only Trough” onto the “Data-Driven Slope”.

The Three Tiers

1. Deterministic evals — pass/fail assertions that extract determinism from the probabilistic system. The most useful kind (per Discord’s Ian Webster): fast, developer-focused.^[raw/aihero/three-types-of-evals.md] Example: Discord’s Clyde bot always replying with a lowercase first letter. Schema conformance from structured outputs is the cheapest deterministic oracle.
2. LLM-as-a-judge — an LLM evaluates outputs against criteria or ground truth (autoevals-style templates: humor, battle/factuality comparison). Treat as a smoke test, not a real test: failing a judge is a good indicator a human should look, but passing one proves little.
3. Human feedback — irreplaceable for long-form generation and certain factuality classes; every LLM app needs some human oversight.

Tooling

Evalite (Vitest-based, by Matt Pocock) exemplifies eval tooling integrated into the dev loop — run alongside unit tests in the same framework. See error analysis and evals and eval driven development for the broader discipline, evals skills for evals as agent skills.

Related

error analysis and evals, eval driven development, generator evaluator loop, structured outputs, evals skills, error analysis and evals.