loading…

Generator-Evaluator Loop

A multi-agent harness pattern, described by prithvi rajasekaran of anthropic’s Labs team, that separates the agent doing the work from the agent judging it. Taking “inspiration from Generative Adversarial Networks (GANs),” the harness pairs a generator agent with a standalone evaluator agent, because agents grading their own output “tend to respond by confidently praising the work—even when, to a human observer, the quality is obviously mediocre.”

The key claim: “tuning a standalone evaluator to be skeptical turns out to be far more tractable than making a generator critical of its own work, and once that external feedback exists, the generator has something concrete to iterate against.”

loop:
  proposal = generator(task, constraints)
  verdict  = evaluator(proposal, criteria)   # independent context
  if verdict.fail: constraints += verdict.failures
  else: return proposal

Making subjective quality gradable

For frontend design — where there is “no binary check equivalent to a verifiable software test” — the harness converts taste into four gradable criteria given to both agents: design quality, originality, craft, and functionality. Design quality and originality were weighted more heavily because Claude scored well on craft and functionality by default; the criteria “explicitly penalized highly generic ‘AI slop’ patterns” such as “purple gradients over white cards.” The evaluator was calibrated with few-shot examples with detailed score breakdowns, and used the Playwright MCP to navigate the live page — screenshotting and studying the implementation — before scoring. Runs went 5 to 15 iterations per generation, with full runs stretching up to four hours.

Criteria wording steered outputs directly: phrases like “the best designs are museum quality” pushed designs toward a particular visual convergence, and even first-iteration outputs beat an unprompted baseline before any evaluator feedback.

The three-agent architecture

Scaled to full-stack coding, the pattern becomes planner → generator → evaluator, built on the Claude Agent SDK:

• Planner — expands “a simple 1-4 sentence prompt” into a full product spec, kept intentionally high-level so spec errors don’t “cascade into the downstream implementation.”
• Generator — works in sprints, one feature at a time, on a React, Vite, FastAPI, and SQLite (later PostgreSQL) stack with git.
• Evaluator — clicks through the running application via Playwright MCP “the way a user would,” grading each sprint against hard thresholds; any criterion below threshold fails the sprint.

Before each sprint the generator and evaluator negotiate a sprint contract — agreeing on what “done” looked like for that chunk of work before any code was written. Contracts were granular — Sprint 3 alone had 27 criteria covering the level editor. Agents communicated via files, a concrete instance of handoff artifacts.

Context resets vs. compaction

The post distinguishes two responses to context rot on long tasks. Compaction summarizes earlier conversation in place so the same agent continues on a shortened history; context resets clear the window entirely and start a fresh agent with a structured handoff. Resets counter “context anxiety” — models “wrapping up work prematurely as they approach what they believe is their context limit” — which compaction alone cannot fix because it “doesn’t give the agent a clean slate.” Claude Sonnet 4.5 exhibited context anxiety strongly enough that resets became essential; Opus 4.5 “largely removed that behavior on its own,” letting the newer harness drop resets and run one continuous session with automatic compaction.

Cost and outcome

On a retro-game-maker prompt, a solo agent ran 20 min for $9; the full harness ran 6 hr for $200 — “over 20x more expensive, but the difference in output quality was immediately apparent.” The planner expanded the one-sentence prompt into a 16-feature spec spread across ten sprints, and the harness build’s play mode actually worked where the solo run’s game was broken.

Independent evaluator context is the evals discipline applied to the proposal loop: the grader is a structured rubric, not vibes. [source: anthropic-demystifying-evals-for-ai-agents-2026]

Failure Modes

Symptom	Root cause	Fix
Generator ignores evaluator feedback	Feedback not injected as structured constraint	Feed failures back as explicit next-iteration constraints
Evaluator rubber-stamps	Same model, same prompt, no independence	Separate evaluator context; different criteria per criterion
Loop never terminates	No exit criteria	Hard cap iterations; escalate remaining disagreements to a human

Rule of Thumb

One loop iteration = one proposal + one evaluation; anything more blurs attribution of what improved.

Related

• agent harness engineering
• multi agent orchestration
• agentic manual testing
• handoff artifacts
• context rot
• error analysis and evals