loading…

Site-as-Eval-Subject Grading

Treating a website (or any shippable artifact) as the subject of its own eval suite: deterministic graders + LLM judges produce a per-column scorecard, and design/content changes are accepted only when they improve the scorecard. Generalizes eval driven development from prompts to shipped interfaces. Grounded in conformance suites as fitness functions: ambiguous “make the site better” becomes a fitness function the change must clear.

Built first as the pyweb.dev grader suite (2026-08-30, PR #107): 6 code graders + 5 planned judges, per-column regression gate.

Architecture

• Grader interface (modeled on Anthropic’s cwc-workshops eval-driven-agent-development, Apache-2.0): Grader{name, kind: "code" | "judge", description, grade(ctx), pass?(v), scale{min,max,good}}. Adding a metric = appending one object.
• Shared GraderContext built once per page (markdown, built HTML, frontmatter, graph degree, screenshot). Graders stay pure functions of the context.
• Independent columns, no blended composite — per agentic code quality: quality has independent acceptance dimensions; a single number hides regressions. The merge gate is per-column: no regressions, targeted column improves.
• Fixtures frozen per climb — the page set + judge settings (temp 0, pinned seed) are immutable for the duration of an optimization loop; changing them restarts the baseline.

First-baseline results (pyweb.dev, 2026-08-30)

Column	Mean	Pass rate
Frontmatter complete	0.81	81%
Code-block density	0.41	41%
Failure modes	0.47	47%
Graph health	7.96	94%
Slop words (mech)	0.03	100%
Dead links	0.00	100%

The baseline immediately drove one content fix (missing frontmatter on 19% of concepts) and one framework fix (below).

Discoveries from building it

1. Machine-readable ground truth lives where the build puts it, not where you expect

The wiki edge graph is written to dist/api/graph.json by the build step — NOT public/api/graph.json. A grader that reads the wrong path silently degrades to zero (empty fallback) and reports 0% graph health — a wrong number that looks like a real finding. Rule: before trusting any deterministic grader’s first run, manually verify one non-trivial value by hand. Zero-feeling results deserve suspicion in both directions.

2. Mechanical slop lists have false positives on technical text

“seamlessly” flagged a sentence describing an adversarial pipeline switching providers — accurate technical description, not slop. “holistic” flagged a legitimate diagnosis of model limitations. Both single-word flags are context-dependent; the mechanical column is a cheap negative filter (its pass = “no slop”) but flags must be human-reviewed before becoming content edits. Pattern-level judges (no-ai-slop style: “binary contrasts”, “throat-clearing openers”) have far lower false-positive rates than word-level lists.

3. The Goodhart surface is per-column

Each grader is independently gameable: stuffing code blocks games code-density; shoehorning “failure” mentions games failure-modes; adding wikilinks games graph health. Mitigations: (a) an objective-adherence judge that checks whether changes serve the actual reader goal; (b) per-column regression gate so gaming one column cannot hide damage in another; (c) calibration against human judgment before trusting judge columns.

4. Independent verification catches what self-report cannot

Adversarial audit of the first smoke report found the slop false positives within minutes — the framework author (the agent) had classified them as 100% pass without reading the flagged contexts. An eval framework built by an agent needs a second agent (or human) to adversarially audit it before its numbers are trusted. designing for verifiability applies to the grader itself: report metric definitions so an auditor can recompute.

Judge best practices (from OpenAI guidance + Hamel Husain + web research)

• Anchored exemplars: include 2-3 canonical snapshots in the judge prompt at fixed scores (a 2/5 “poor hierarchy” example, a 5/5 gold standard) so the judge calibrates against references, not vibes.
• CoT-before-score; structured outputs (zod strict) for machine-grading; numeric score AND pass/fail threshold per column.
• Pairwise comparison with position order swapped twice cancels position bias when judging a redesign against the current site.
• Judge noise check: two runs of the same fixture must agree within tolerance before any climb starts.

Adopted from independent web research (2026-08-30)

1. Mutation testing of judges (OpenAI): deliberately inject defects (broken links, low-contrast headings, malformed metadata) and assert each judge’s score drops. An insensitive judge prompt is rewritten, not trusted.
2. Multi-persona panel: decompose design judging into orthogonal personas (information architect, first-time reader, accessibility auditor) aggregated by weighted consensus, reducing bias toward generic aesthetic tropes.
3. Holdout split: hill-climb against a training subset of pages; block release if the holdout set fails to generalize. Generalization beats fixture-fit.
4. Grounded evidence locators: judges must cite exact text snippets / DOM selectors / bounding boxes for every deduction; deductions without evidence are discarded.
5. Discrete coarse rubrics (1-4 with strict per-step criteria, not 1-100 floats) + N=3 majority voting on borderline scores to fight non-determinism on edge cases.
6. Known failure modes to watch: LLM aesthetic homogenization (all judged sites converge to generic AI style) - separate stylistic grading from structural utility; viewport truncation blindspots - pair full-page screenshots with DOM/accessibility dumps.

Novel wiki ideas harvested

• Dual-stream multimodal verification: computed DOM accessibility trees
  • multi-viewport rendered snapshots produce hallucination-proof site judges.
• Evolutionary anti-regression invariant mining: automatically promote repeated judge findings into deterministic code graders, cutting judge token cost over time (e.g., after the slop judge flags a pattern three runs, add it to the mechanical slop-words list).

Related

eval taxonomy (three tiers), conformance suites as fitness functions (the climb loop), agentic code quality (independent dimensions), eval driven development (error analysis first), designing for verifiability, evals skills (graders as skills), generator evaluator loop.