---
title: "Conformance Suites as Fitness Functions"
description: "Pairing agents with uncompromising deterministic test suites so they act as autonomous optimization engines instead of drifting generators."
section: "concepts"
type: "concept"
created: "2026-08-24"
updated: "2026-08-24"
confidence: "high"
tags: ["agents", "evaluation", "tdd", "workflow", "principle"]
canonical: "https://pyweb.dev/wiki/conformance-suites-as-fitness-functions"
---
# Conformance Suites as Fitness Functions
**Conformance Suites as Fitness Functions** is the breakthrough pattern that transforms agents from drifting generators into autonomous optimization engines. When agents are paired with uncompromising, deterministic test suites, they can grind complex optimization tasks into working code without human intervention.
## The Core Thesis
**The Problem:** When agents receive ambiguous tasks like "make this better," they hallucinate and drift without objective success criteria.
**The Solution:** Language-independent conformance test suites provide deterministic, objective fitness functions that enable autonomous optimization loops.
```mermaid
flowchart TD
AR["AGENT AUTORESEARCH / REWRITE"] --> PE["Propose Code Edit / Patch"]
PE --> CS["CONFORMANCE SUITE
(Objective assertions)"]
CS -->|Exit Code != 0| FAIL["Feed Failure Output
Back into Agent Loop"]
CS -->|Exit Code == 0| PASS["Commit & Advance to
Next Feature / Goal"]
FAIL --> AR
```
## Landmark Case Studies
### Bun Zig→Rust Rewrite (July 2026)
The most dramatic validation of conformance-driven development came from [Jarred Sumner's rewriting of Bun's runtime engine](https://bun.com/blog/bun-in-rust) from **Zig to safe Rust**:
**The Enabling Factor:** Bun's extensive test suite was written in **TypeScript**, completely external to the implementation language. This language-independent test suite served as an objective conformance suite for the new Rust engine.
**Execution pattern:** Parallel agent loops worked against the TypeScript suite while humans monitored the workflow and reviewed the process rather than attempting a conventional line-by-line review of the generated port. The published case study includes detailed token, duration, and performance measurements. [[source: agentic-engineering-trends-2026-synthesis]](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)
### Shopify Liquid Optimization (March 2026)
[Tobias Lütke's performance PR](https://github.com/Shopify/liquid/pull/2056) demonstrated the **Autoresearch Pattern**:
**Setup:** A durable prompt, a benchmark script, and the project's unit tests
**Architecture:** Pi agent ran in autonomous overnight loop, proposing micro-optimizations, benchmarking throughput, and recording state in `autoresearch.jsonl`
**Result:** A long sequence of benchmark-gated micro-optimizations, with exact measurements preserved in the linked pull request. [[source: agentic-engineering-trends-2026-synthesis]](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)
### "Vibe Porting" - Rapid Cross-Language Rewrites
**JSONata to Go (Reco AI):** [Ported the JSONata query engine](https://www.reco.ai/blog/we-rewrote-jsonata-with-ai) from JavaScript to native Go, using the existing test suite and shadow production traffic as verification.
## Engineering Principles
### 1. Language Independence
Test suites must be external to implementation language to enable cross-language ports and rewrites without losing verification capability.
### 2. Deterministic Success Criteria
Agents need binary pass/fail signals, not subjective quality judgments. Exit codes (0 = success, non-zero = failure) provide unambiguous feedback.
### 3. Comprehensive Coverage
Thousand-assertion test suites catch edge cases that humans miss during manual review of generated code.
### 4. Granular Feedback Loops
Failed test outputs feed directly back into agent prompts, creating tight optimization cycles around specific failure modes.
## Anti-Patterns
**Conformance Theater:** Creating tests that are too permissive or don't exercise critical code paths leads to false confidence in generated code quality.
**Suite Drift:** Test suites that aren't maintained in parallel with specification evolution become obsolete fitness functions.
## Related Concepts
- [red green tdd](/wiki/red-green-tdd)
- [tdd with agents](/wiki/tdd-with-agents)
- [automated eval engineering](/wiki/automated-eval-engineering)
- [agent harness engineering](/wiki/agent-harness-engineering)
- [multi agent orchestration](/wiki/multi-agent-orchestration)
---
## Agent Navigation
cluster: person (170 pages) | betweenness: 170.2
### References (outbound)
- [Red/Green TDD](https://pyweb.dev/wiki/red-green-tdd.md)
- [TDD With Agents](https://pyweb.dev/wiki/tdd-with-agents.md)
- [Automated Eval Engineering](https://pyweb.dev/wiki/automated-eval-engineering.md)
- [Agent Harness Engineering](https://pyweb.dev/wiki/agent-harness-engineering.md)
- [Multi-Agent Orchestration](https://pyweb.dev/wiki/multi-agent-orchestration.md)
### Referenced by (inbound)
- [Agentic Software Factory](https://pyweb.dev/wiki/agentic-software-factory.md)
- [Agentic Vulnerability Lifecycle](https://pyweb.dev/wiki/agentic-vulnerability-lifecycle.md)
- [Multi-Agent Orchestration](https://pyweb.dev/wiki/multi-agent-orchestration.md)
- [Red/Green TDD](https://pyweb.dev/wiki/red-green-tdd.md)
- [Releasable Patch Rate](https://pyweb.dev/wiki/releasable-patch-rate.md)
- [Site-as-Eval-Subject Grading](https://pyweb.dev/wiki/site-as-eval-subject-grading.md)
- [Andrej Karpathy](https://pyweb.dev/wiki/andrej-karpathy.md)
- [Jarred Sumner](https://pyweb.dev/wiki/jarred-sumner.md)
- [Lalit Maganti](https://pyweb.dev/wiki/lalit-maganti.md)
### Evidence (verified primary sources)
- [agentic-engineering-trends-2026-synthesis](https://pyweb.dev/wiki/raw/articles/agentic-engineering-trends-2026-synthesis.md) | origin: https://pyweb.dev/wiki/raw/articles/agentic-engineering-trends-2026-synthesis | ingested: 2026-08-24 | sha256: 6c78ca873a4df069f1606e9f68538de7b428633720e48e184fad0ea31b5d2a25
### Machine endpoints
- Knowledge graph: https://pyweb.dev/api/graph.json
- Graph analysis: https://pyweb.dev/api/graph-analysis.json
- Context index: https://pyweb.dev/llms.txt