---
title: "Site-as-Eval-Subject Grading"
description: "Grading a static site with deterministic graders and LLM judges, then hill-climbing the design against the scorecard."
section: "concepts"
type: "concept"
created: "2026-08-30"
updated: "2026-08-30"
confidence: "high"
tags: ["evaluation", "agents", "technique", "workflow"]
canonical: "https://pyweb.dev/wiki/site-as-eval-subject-grading"
---
# Site-as-Eval-Subject Grading

Treating a website (or any shippable artifact) as the **subject of its own
eval suite**: deterministic graders + LLM judges produce a per-column
scorecard, and design/content changes are accepted only when they improve the
scorecard. Generalizes [eval driven development](/wiki/eval-driven-development) from prompts to shipped
interfaces. Grounded in [conformance suites as fitness functions](/wiki/conformance-suites-as-fitness-functions): ambiguous
"make the site better" becomes a fitness function the change must clear.

Built first as the pyweb.dev grader suite (2026-08-30, PR #107): 6 code
graders + 5 planned judges, per-column regression gate.

## Architecture

- **Grader interface** (modeled on Anthropic's cwc-workshops
  eval-driven-agent-development, Apache-2.0): `Grader{name, kind: "code" |
  "judge", description, grade(ctx), pass?(v), scale{min,max,good}}`. Adding a
  metric = appending one object.
- **Shared GraderContext** built once per page (markdown, built HTML,
  frontmatter, graph degree, screenshot). Graders stay pure functions of the
  context.
- **Independent columns, no blended composite** — per [agentic code quality](/wiki/agentic-code-quality):
  quality has independent acceptance dimensions; a single number hides
  regressions. The merge gate is per-column: no regressions, targeted column
  improves.
- **Fixtures frozen per climb** — the page set + judge settings (temp 0,
  pinned seed) are immutable for the duration of an optimization loop;
  changing them restarts the baseline.

## First-baseline results (pyweb.dev, 2026-08-30)

| Column | Mean | Pass rate |
|---|---|---|
| Frontmatter complete | 0.81 | 81% |
| Code-block density | 0.41 | 41% |
| Failure modes | 0.47 | 47% |
| Graph health | 7.96 | 94% |
| Slop words (mech) | 0.03 | 100% |
| Dead links | 0.00 | 100% |

The baseline immediately drove one content fix (missing frontmatter on 19%
of concepts) and one framework fix (below).

## Discoveries from building it

### 1. Machine-readable ground truth lives where the build puts it, not where you expect
The wiki edge graph is written to `dist/api/graph.json` by the build step —
NOT `public/api/graph.json`. A grader that reads the wrong path silently
degrades to zero (empty fallback) and reports **0% graph health** — a wrong
number that looks like a real finding. **Rule: before trusting any
deterministic grader's first run, manually verify one non-trivial value by
hand.** Zero-feeling results deserve suspicion in both directions.

### 2. Mechanical slop lists have false positives on technical text
"seamlessly" flagged a sentence describing an adversarial pipeline switching
providers — accurate technical description, not slop. "holistic" flagged a
legitimate diagnosis of model limitations. Both single-word flags are
**context-dependent**; the mechanical column is a cheap negative filter (its
pass = "no slop") but flags must be human-reviewed before becoming content
edits. Pattern-level judges (no-ai-slop style: "binary contrasts",
"throat-clearing openers") have far lower false-positive rates than
word-level lists.

### 3. The Goodhart surface is per-column
Each grader is independently gameable: stuffing code blocks games
code-density; shoehorning "failure" mentions games failure-modes; adding
wikilinks games graph health. Mitigations: (a) an objective-adherence judge
that checks whether changes serve the actual reader goal; (b) per-column
regression gate so gaming one column cannot hide damage in another; (c)
calibration against human judgment before trusting judge columns.

### 4. Independent verification catches what self-report cannot
Adversarial audit of the first smoke report found the slop false positives
within minutes — the framework author (the agent) had classified them as
100% pass without reading the flagged contexts. **An eval framework built by
an agent needs a second agent (or human) to adversarially audit it before
its numbers are trusted.** [designing for verifiability](/wiki/designing-for-verifiability) applies to the
grader itself: report metric definitions so an auditor can recompute.

## Judge best practices (from OpenAI guidance + Hamel Husain + web research)

- **Anchored exemplars**: include 2-3 canonical snapshots in the judge prompt
  at fixed scores (a 2/5 "poor hierarchy" example, a 5/5 gold standard) so
  the judge calibrates against references, not vibes.
- CoT-before-score; structured outputs (zod strict) for machine-grading;
  numeric score AND pass/fail threshold per column.
- Pairwise comparison with position order swapped twice cancels position
  bias when judging a redesign against the current site.
- Judge noise check: two runs of the same fixture must agree within
  tolerance before any climb starts.

### Adopted from independent web research (2026-08-30)

1. **Mutation testing of judges** (OpenAI): deliberately inject defects
   (broken links, low-contrast headings, malformed metadata) and assert each
   judge's score drops. An insensitive judge prompt is rewritten, not
   trusted.
2. **Multi-persona panel**: decompose design judging into orthogonal
   personas (information architect, first-time reader, accessibility
   auditor) aggregated by weighted consensus, reducing bias toward generic
   aesthetic tropes.
3. **Holdout split**: hill-climb against a training subset of pages; block
   release if the holdout set fails to generalize. Generalization beats
   fixture-fit.
4. **Grounded evidence locators**: judges must cite exact text snippets / DOM
   selectors / bounding boxes for every deduction; deductions without
   evidence are discarded.
5. **Discrete coarse rubrics** (1-4 with strict per-step criteria, not
   1-100 floats) + N=3 majority voting on borderline scores to fight
   non-determinism on edge cases.
6. **Known failure modes to watch**: LLM aesthetic homogenization (all
   judged sites converge to generic AI style) - separate stylistic grading
   from structural utility; viewport truncation blindspots - pair full-page
   screenshots with DOM/accessibility dumps.

### Novel wiki ideas harvested

- **Dual-stream multimodal verification**: computed DOM accessibility trees
  + multi-viewport rendered snapshots produce hallucination-proof site
  judges.
- **Evolutionary anti-regression invariant mining**: automatically promote
  repeated judge findings into deterministic code graders, cutting judge
  token cost over time (e.g., after the slop judge flags a pattern three
  runs, add it to the mechanical slop-words list).

## Related

[eval taxonomy](/wiki/eval-taxonomy) (three tiers), [conformance suites as fitness functions](/wiki/conformance-suites-as-fitness-functions)
(the climb loop), [agentic code quality](/wiki/agentic-code-quality) (independent dimensions),
[eval driven development](/wiki/eval-driven-development) (error analysis first),
[designing for verifiability](/wiki/designing-for-verifiability), [evals skills](/wiki/evals-skills) (graders as skills),
[generator evaluator loop](/wiki/generator-evaluator-loop).

---

## Agent Navigation

cluster: person (170 pages) | betweenness: 126.6

### References (outbound)
- [Eval-Driven Development](https://pyweb.dev/wiki/eval-driven-development.md)
- [Conformance Suites as Fitness Functions](https://pyweb.dev/wiki/conformance-suites-as-fitness-functions.md)
- [Agentic Code Quality](https://pyweb.dev/wiki/agentic-code-quality.md)
- [Designing for Verifiability](https://pyweb.dev/wiki/designing-for-verifiability.md)
- [Eval Taxonomy](https://pyweb.dev/wiki/eval-taxonomy.md)
- [Generator-Evaluator Loop](https://pyweb.dev/wiki/generator-evaluator-loop.md)

### Evidence (verified primary sources)
- [hamel-husain-ai-evals-faq-2026](https://pyweb.dev/wiki/raw/articles/hamel-husain-ai-evals-faq-2026.md) | origin: https://hamel.dev/blog/posts/evals-faq/ | ingested: 2026-08-27 | sha256: fe8dd2419e8d96756f494a707be985227ae5724da945c9c4dcede5ca03f89154
- [airbnb-eval-driven-development-2026](https://pyweb.dev/wiki/raw/articles/airbnb-eval-driven-development-2026.md) | origin: https://medium.com/airbnb-engineering/eval-driven-development-lessons-from-evaluating-genai-at-scale-e817e5ae5788 | ingested: 2026-08-27 | sha256: 94414a24787cabca4537e9a34113139762e66eb17d3996269377298569a51b0c

### Machine endpoints
- Knowledge graph: https://pyweb.dev/api/graph.json
- Graph analysis: https://pyweb.dev/api/graph-analysis.json
- Context index: https://pyweb.dev/llms.txt