# pyweb.dev Full Context Index
> Complete knowledge base and resource library for autonomous AI agents and engineering systems.
> Pre-rendered flat markdown context index covering AI engineering, agentic architecture, and verifiable patterns.

# Knowledge Structure (Graph Overview)
> Top-level topology computed from graph analysis: 181 nodes, 3 topical clusters, 0 orphans.

## Topical Clusters
- **person** (170 pages, density: 0.04): agent-containment-and-blast-radius, agent-harness-engineering, agent-native-infrastructure, agentic-code-quality, agentic-engineering-patterns
- **llm-fundamentals** (8 pages, density: 0.54): agents-vs-workflows, llm-app-improvement-ladder, llm-message-protocol, model-provider-abstraction, structured-outputs
- **agents** (3 pages, density: 1): ag-ui-protocol, claude-managed-agents, copilotkit

## Top Broker Pages (High Betweenness)
- [[agentic-code-quality]] (betweenness score: 7322)
- [[agent-harness-engineering]] (betweenness score: 3548.7)
- [[agentic-quality-evidence]] (betweenness score: 2060.3)
- [[agentic-engineering-patterns]] (betweenness score: 1735.1)
- [[clean-architecture]] (betweenness score: 1624.5)

## Structural Gaps & Research Questions
- *llm-fundamentals ↔ agents*: How do concepts in llm-fundamentals (agents-vs-workflows, llm-app-improvement-ladder) bridge into agents (ag-ui-protocol, claude-managed-agents)?

---


# Section: CONCEPTS

## AG-UI Protocol
- URL: https://pyweb.dev/wiki/ag-ui-protocol
- Raw Markdown: https://pyweb.dev/wiki/ag-ui-protocol.md
- Type: concept
- Summary: Open, lightweight, event-based standard connecting AI agent runtimes to user-facing frontend applications.
- Tags: agents, workflow, context-engineering, subagents

# AG-UI Protocol

**AG-UI (Agent–User Interaction Protocol)** is an open, event-based specification that standardizes communication between backend agent execution loops and user-facing frontend applications. Originated by [copilotkit](/wiki/copilotkit) in partnership with ecosystem frameworks (LangChain, CrewAI, Microsoft Agent Framework, AWS Bedrock AgentCore), AG-UI defines the frontend interaction layer alongside MCP (tools) and A2A (agent coordination). [[source: ag-ui-protocol-specification-2026]](/wiki/raw/articles/ag-ui-protocol-specification-2026)

## The Three Agentic Protocols

```mermaid
graph LR
    User[User / Frontend UI] <-- "AG-UI (Agent-User Interaction)" --> Agent[Agent Runtime / Backend]
    Agent <-- "MCP (Model Context Protocol)" --> Tools[External Tools & Data]
    Agent <-- "A2A (Agent to Agent)" --> Subagents[Distributed Agents]
```

| Layer | Protocol | Primary Origin / Backers | Purpose |
| :--- | :--- | :--- | :--- |
| **Agent ↔ User** | **AG-UI** | [copilotkit](/wiki/copilotkit), LangChain, AWS, Microsoft | Standardizes streaming state, generative UI, client tools, and human-in-the-loop gates. |
| **Agent ↔ Tools & Data** | **MCP** | [anthropic](/wiki/anthropic) | Standardizes tool definitions, context injection, and resource prompts. |
| **Agent ↔ Agent** | **A2A** | Google | Standardizes distributed agent discovery, delegation, and message routing. |

 [[source: ag-ui-protocol-specification-2026]](/wiki/raw/articles/ag-ui-protocol-specification-2026)

## Protocol Primitives & Lifecycle Events

AG-UI models agent execution as a stateful, bi-directional event stream transported over Server-Sent Events (SSE) or WebSockets:

1. **Lifecycle & Text Streaming:** `RUN_STARTED`, `TEXT_MESSAGE_START`, `TEXT_MESSAGE_CONTENT`, `TEXT_MESSAGE_END`, `RUN_FINISHED`.
2. **Tool Execution Tracing:** `TOOL_CALL_START`, `TOOL_CALL_ARGS`, `TOOL_CALL_END`, `TOOL_CALL_RESULT`.
3. **Reasoning & Thinking:** `REASONING_START`, `REASONING_DELTA`, `REASONING_END` to display progress signals without leaking raw chain-of-thought tokens.
4. **Shared State Sync:** Streamed event-sourced snapshots and delta updates (`STATE_SNAPSHOT`, `STATE_DELTA`) between client state and agent memory.
5. **Human-in-the-Loop (HITL) Interrupts:** Protocol-level pause, approve, reject, or edit capabilities enabling gated tool execution.

## Architectural Patterns

### 1. Pure Event-to-Message Fold
Frontends (such as assistant-ui or CopilotKit) maintain a pure reducer that projects the session event log into the client message model. Replaying stored historical events and tailing live SSE streams execute through the exact same fold function, ensuring state consistency across page reloads. [[source: assistant-ui-claude-managed-agents-2026]](/wiki/raw/articles/assistant-ui-claude-managed-agents-2026)

### 2. Generative UI (Static & Declarative)
Rather than raw markdown alone, agent backends emit structured intents that client components mount as typed, interactive widgets (charts, forms, parameter selectors). [[source: ag-ui-protocol-specification-2026]](/wiki/raw/articles/ag-ui-protocol-specification-2026)

### 3. Multi-Surface Transport
Because AG-UI defines an event stream rather than a DOM rendering engine, it operates across React, React Native (native mobile components), Angular, and messaging platforms (Slack, Microsoft Teams) via channel adapters. [[source: copilotkit-cma-agui-2026]](/wiki/raw/articles/copilotkit-cma-agui-2026)

## Cross-links
- [agent native infrastructure](/wiki/agent-native-infrastructure) — underlying platform and runtime tier
- [claude managed agents](/wiki/claude-managed-agents) — cloud-hosted agent loop implementing AG-UI adapters
- [copilotkit](/wiki/copilotkit) — originating organization
- [anthropic](/wiki/anthropic) — MCP and CMA ecosystem partner

---

## Agent Containment and Blast Radius
- URL: https://pyweb.dev/wiki/agent-containment-and-blast-radius
- Raw Markdown: https://pyweb.dev/wiki/agent-containment-and-blast-radius.md
- Type: concept
- Summary: Capping what an agent can do — not just what it tends to do — via sandboxes, VMs, egress controls, and classifier-gated approvals layered over fallible human oversight.
- Tags: agents, security, workflow, evaluation

# Agent Containment and Blast Radius

As agents grow more capable, the risk of a deployment has two components: how likely a failure is, and how much damage one could do. Model training and safeguards drive down the first; the second — the theoretical **blast radius** — only grows as capabilities and access expand. The engineering question becomes how to cap it. [[source: anthropic-engineering-how-we-contain-claude-2026]](/wiki/raw/articles/anthropic-engineering-how-we-contain-claude-2026)

## Why human-in-the-loop supervision fails

Per-action permission prompts theoretically keep users safe, but Anthropic's telemetry showed users approve roughly 93% of permission prompts. The more approvals a user sees, the less attention they pay to each — **approval fatigue** turns an oversight feature into rubber-stamping. Anonymized usage data showed experienced users auto-approve roughly twice as often as new users, supervising the agent only when it visibly drifts. [[source: anthropic-engineering-how-we-contain-claude-2026]](/wiki/raw/articles/anthropic-engineering-how-we-contain-claude-2026)

## Three risk types, three defense components

Risks fall into three categories: **user misuse**, **model misbehavior** (overeager initiative, honest mistakes about blast radius), and **external attackers** (prompt injection and conventional attacks on the runtime). Defenses apply to three components: [[source: anthropic-engineering-how-we-contain-claude-2026]](/wiki/raw/articles/anthropic-engineering-how-we-contain-claude-2026)

1. **The environment** — process sandboxes, VMs, filesystem boundaries, egress controls. Hard boundaries: if credentials never enter the sandbox, they can't be exfiltrated regardless of cause. A tight perimeter lets you relax oversight.
2. **The model** — system prompts, classifiers, probes, training. Probabilistic: shapes what the agent *tends* to do, never what it *can* do.
3. **The external content** — MCP servers, plugins, web tools feed unaudited data into context; an audited connector is not audited data.

## Classifier-gated autonomy (auto mode)

Claude Code auto mode replaces per-action human approval with layered model-based classifiers: a server-side **prompt-injection probe** screens tool outputs entering context, and a two-stage **transcript classifier** (fast single-token filter, then chain-of-thought reasoning only on flagged actions) gates tool calls before execution. The classifier is deliberately **reasoning-blind**: it sees only user messages and tool calls, with assistant text and tool outputs stripped so the agent cannot talk it into a bad call and injected content never reaches it. [[source: anthropic-engineering-claude-code-auto-mode-2026]](/wiki/raw/articles/anthropic-engineering-claude-code-auto-mode-2026)

Measured trade-off: the full pipeline cut false positives on real traffic from 8.5% to 0.4%, at the cost of a 17% false-negative rate on real overeager actions. Anthropic frames this honestly: an improvement over skipping permissions entirely, arguably a regression versus careful manual review. [[source: anthropic-engineering-claude-code-auto-mode-2026]](/wiki/raw/articles/anthropic-engineering-claude-code-auto-mode-2026)

## Containment patterns by audience

- **Ephemeral container** (claude.ai): server-side gVisor container, per-session filesystem — minimal blast radius, low capability ceiling.
- **Human-in-the-loop sandbox** (Claude Code): works only because users are developers who can read bash; an OS-level sandbox (Seatbelt/bubblewrap, network denied by default) cut permission prompts by 84%.
- **Local VM** (Cowork): for non-technical users the boundary must be absolute and always-on — workspace-only mounts, credentials never enter the guest. [[source: anthropic-engineering-how-we-contain-claude-2026]](/wiki/raw/articles/anthropic-engineering-how-we-contain-claude-2026)

## Lessons from failures

- **Everything before the trust dialog is attack surface:** project-local config was parsed before the "Do you trust this folder?" prompt, letting a committed hook execute on open. Treat project-open and config-load like inbound internet requests.
- **The user is an injection vector:** a phished "run this for me" prompt exfiltrated credentials in 24 of 25 attempts — model-layer defenses anchor on user intent, so only environment controls (egress, filesystem boundaries) hold.
- **An allowlist is a capability grant, not a destination filter:** allowing `api.anthropic.com` allowed file uploads to an attacker's account through it.
- **The weakest layer is the one you built yourself:** hardened primitives (gVisor, seccomp, hypervisors) held; the custom proxy broke. [[source: anthropic-engineering-how-we-contain-claude-2026]](/wiki/raw/articles/anthropic-engineering-how-we-contain-claude-2026)

## Related Concepts
- [agent harness engineering](/wiki/agent-harness-engineering)
- [multi agent orchestration](/wiki/multi-agent-orchestration)
- [subagents and context management](/wiki/subagents-and-context-management)
- [agentic engineering patterns](/wiki/agentic-engineering-patterns)
- [anthropic](/wiki/anthropic)

---

## Agent Harness Engineering
- URL: https://pyweb.dev/wiki/agent-harness-engineering
- Raw Markdown: https://pyweb.dev/wiki/agent-harness-engineering.md
- Type: concept
- Summary: Designing the runtime infrastructure around an agent's model — sandboxes, tool contracts, progressive disclosure, and verification gates — as the primary driver of reliability.
- Tags: agents, context-engineering, subagents, workflow, evaluation

# Agent Harness Engineering

**Agent Harness Engineering** is the discipline of designing, constraining, and operating the runtime infrastructure that surrounds an AI agent's core model. Rather than focusing on prompt tweaks or monolithic framework abstractions, harness engineering treats the environment around the model — execution sandboxes, tool contracts, progressive context loading, deterministic feedback loops, and verification gates — as the primary driver of agent reliability and task completion.

The paradigm shifts the software engineering problem from *"How do we prompt the model to do X?"* to *"How do we build an operating substrate where failure is caught early, state is durable, and actions are verified mechanically?"*

## The 9 Functional Pillars of an Agent Harness

Across frontier agent implementations and literature, an agent harness decomposes into 9 structural domains:

```mermaid
flowchart TD
    subgraph Pillars["9 Pillars of Agent Harness Architecture"]
        P1["1. Orchestration & Loops (Worktrees, swarms)"]
        P2["2. Context & State (AST symbol graphs)"]
        P3["3. Execution & Sandboxes (Containers, CDP)"]
        P4["4. Tool Contracts (MCP, JSON-RPC)"]
        P5["5. Progressive Disclosure (Agent Skills)"]
        P6["6. Observability (Distributed tracing)"]
        P7["7. Guardrails & Blast Radius (Egress, gates)"]
        P8["8. Evals & Red-Teaming (SWE-bench)"]
        P9["9. Reference Implementations (CLI/TUI)"]
    end
```

### 1. Harness Over Framework
Standard agent frameworks often introduce rigid, high-latency abstractions around prompt chains. Harness engineering favors minimal, durable runtimes that manage event-log persistence, stateless tool replay, and session resumability across network boundaries.

### 2. Sandbox Isolation & Ephemeral Worktrees
Production harnesses decouple the agent's reasoning loop from the host environment:
- **Filesystem Boundaries:** Executing inside Git worktrees or container sandboxes prevents workspace pollution during exploratory edits.
- **Egress & Token Governance:** Network restrictions, restricted OS tokens, and command runners mitigate prompt injection and approval fatigue.

### 3. Progressive Capability Disclosure
Stuffing full documentation and all tool definitions into the root prompt causes [prompt bloat](/wiki/prompt-bloat) and degrades reasoning sharpness in the [smart zone](/wiki/smart-zone). Harnesses expose compact tool/skill indexes (~50 characters per trigger) and hydrate full procedural markdown instructions (`SKILL.md`) only when invoked by the agent.

### 4. Deterministic Quality Gates
Harnesses enforce proof-of-work before declaring success:
- **Red-Green Verification:** Requiring failing reproduction tests before applying fixes ([red green tdd](/wiki/red-green-tdd)).
- **Mechanical Validation:** Automatic syntax checks, type checking, and test suites run within the harness loop to feed immediate compiler feedback back to the agent.

### 5. AST & Graph Context Over Raw Embeddings
Naïve RAG and full-directory dumps flood token windows. State-of-the-art context harnesses build local AST symbol graphs and hierarchical memory stores ([context engineering](/wiki/context-engineering)), allowing agents to query precise interface definitions and cross-references on demand.

## Why Harness Work Became Durable (2026)

[drew breunig](/wiki/drew-breunig) marks the economic turn: prior to Fable it felt silly to invest heavily in your coding harness or context strategies, because "A new model would arrive at the same price (or cheaper!) and paper over most of your problems." Once frontier capability stopped arriving at flat prices, teams "started to think about what work went where" — harness and context investment stopped being throwaway glue. [[source: simon-willison-quoting-drew-breunig-2026]](/wiki/raw/articles/simon-willison-quoting-drew-breunig-2026)

[hamel husain](/wiki/hamel-husain) adds that a large portion of the harness is data science: beyond tests and specifications, production harnesses include an observability stack — logs, metrics, and traces exposed to the agent so it can tell when it is going off track. [[source: hamel-husain-the-revenge-of-the-data-scientist-2026]](/wiki/raw/articles/hamel-husain-the-revenge-of-the-data-scientist-2026)

## Rule of Thumb

Every capability the model could abuse must pass through a gate the
harness owns; the model proposes, the harness disposes.

## Failure Modes

| Symptom | Root cause | Fix |
|---|---|---|
| Model behavior escapes sandbox | Capabilities granted beyond task needs | Least-privilege tool grants per session |
| Harness swallows agent errors | Catch-all logging without routing | Errors surface as structured feedback to the agent |

## Related Concepts
- [agentic code quality](/wiki/agentic-code-quality)
- [agentic software factory](/wiki/agentic-software-factory)
- [agentic engineering entity accounting](/wiki/agentic-engineering-entity-accounting)
- [clean architecture](/wiki/clean-architecture)
- [dependency rule](/wiki/dependency-rule)
- [codex harness architecture](/wiki/codex-harness-architecture)
- [agent containment and blast radius](/wiki/agent-containment-and-blast-radius)
- [deepseek harness](/wiki/deepseek-harness)
- [cordis framework](/wiki/cordis-framework)
- [agent native infrastructure](/wiki/agent-native-infrastructure)
- [context engineering](/wiki/context-engineering)
- [progressive disclosure](/wiki/progressive-disclosure)
- [red green tdd](/wiki/red-green-tdd)
- [subagents and context management](/wiki/subagents-and-context-management)
- [prompt bloat](/wiki/prompt-bloat)
- [smart zone](/wiki/smart-zone)
- [tdd with agents](/wiki/tdd-with-agents)

---

## Agent-Native Infrastructure
- URL: https://pyweb.dev/wiki/agent-native-infrastructure
- Raw Markdown: https://pyweb.dev/wiki/agent-native-infrastructure.md
- Type: concept
- Summary: Full-stack platforms designed around machine cognition, deterministic state machines, and cryptographic agent identity rather than human developer tools.
- Tags: agents, context-engineering, subagents, knowledge-management, workflow

# Agent-Native Infrastructure

**Agent-Native Infrastructure** represents the architectural transition from treating AI coding agents as external automation bolted onto human developer tools to building full-stack platforms designed around machine cognition, deterministic state machines, and cryptographic identity.

Across runtimes, browsers, memory stores, and team collaboration environments, the developer tooling stack is undergoing a fundamental redesign for autonomous agent execution.

## Core Architectural Pillars

### 1. Identity-Scoped Agents as First-Class Teammates
Rather than executing actions under shared human credentials or opaque API keys, modern systems treat agents as distinct organizational entities. Environments like Block Buzz model human and AI collaboration inside shared rooms backed by immutable Nostr relays, granting agents dedicated cryptographic keypairs and verifiable event logs. [[source: block-buzz-2026]](/wiki/raw/articles/block-buzz-2026)

### 2. Standardized Agent Interaction Protocols (AG-UI, MCP, A2A)
As agent runtimes mature, ad-hoc API wiring is replaced by specialized protocol layers: [ag ui protocol](/wiki/ag-ui-protocol) for event-driven frontend streaming and generative UI, MCP for tool connections, and A2A for distributed multi-agent delegation. [[source: ag-ui-protocol-specification-2026]](/wiki/raw/articles/ag-ui-protocol-specification-2026)

### 3. Tiered and Progressive Context Disclosure
Dumping extensive project state into an LLM's system prompt causes prompt bloat and severe attention dilution. Agent-native databases like Volcengine's OpenViking and TencentDB Agent Memory introduce structured hierarchies (L0 abstracts, L1 overviews, L2/L3 deep symbols and traces). Agents explore context deterministically using filesystem primitives (`ls`, `tree`, `find`) and on-demand retrieval rather than black-box vector search. [[source: volcengine-openviking-2026]](/wiki/raw/articles/volcengine-openviking-2026) [[source: tencentdb-agent-memory-2026]](/wiki/raw/articles/tencentdb-agent-memory-2026)

### 4. Portable Units of Capability (Skills & Plugins)
Capabilities are modularized into declarative manifests and procedural routines (`SKILL.md` bundles, `.mdc` workspace rules, and MCP definitions). Standardizing these capabilities into versioned, auditable bundles allows agents to dynamically load procedural skills on demand. [[source: cursor-plugins-marketplace-2026]](/wiki/raw/articles/cursor-plugins-marketplace-2026)

### 5. Code Execution Over CLI Round-Trips
Interacting with tools via iterative, multi-step CLI commands incurs significant latency and context overhead. Agent-native browsers such as Citrolabs ego-lite demonstrate that allowing agents to execute composed JavaScript directly inside the target execution environment achieves 2.5x speed improvements and richer state inspection compared to sequential tool calls. [[source: citrolabs-ego-lite-2026]](/wiki/raw/articles/citrolabs-ego-lite-2026)

### 6. Hosted Runtimes & Isolated Sandboxes
Platforms such as [claude managed agents](/wiki/claude-managed-agents) separate reasoning engines from isolated Linux microVM sandboxes, persisting append-only event trees and enabling human-in-the-loop permission gates. [[source: anthropic-claude-managed-agents-overview-2026]](/wiki/raw/articles/anthropic-claude-managed-agents-overview-2026)

## Related Concepts
- [ag ui protocol](/wiki/ag-ui-protocol)
- [claude managed agents](/wiki/claude-managed-agents)
- [agent harness engineering](/wiki/agent-harness-engineering)
- [context engineering](/wiki/context-engineering)
- [progressive disclosure](/wiki/progressive-disclosure)
- [llm wiki pattern](/wiki/llm-wiki-pattern)
- [subagents and context management](/wiki/subagents-and-context-management)

---

## Agentic Code Quality
- URL: https://pyweb.dev/wiki/agentic-code-quality
- Raw Markdown: https://pyweb.dev/wiki/agentic-code-quality.md
- Type: concept
- Summary: Risk-conditioned verification and authorization controls for autonomous software development.
- Tags: coding-guidelines, feedback-loops, tdd, workflow, evaluation, security

# Agentic Code Quality

Agentic code quality is usefully modeled as a property of the **model–scaffold–toolchain–policy system**, not the generator alone. While practitioners emphasize patterns like [agentic engineering patterns](/wiki/agentic-engineering-patterns) and [eval driven development](/wiki/eval-driven-development), empirical research reveals distinct failure modes: test-oracle defects, reward hacking, scope expansion, noisy automated review, and long-horizon degradation. [agentic quality evidence](/wiki/agentic-quality-evidence) separates observed evidence from the architecture proposed here.

## Quality Vector

A software factory should report independent acceptance dimensions plus their joint intersection:

| Dimension | Direct evidence | Proxy to distrust |
|---|---|---|
| Requirement correctness | Hidden behavioral tests plus expert adjudication | Visible tests alone |
| Regression safety | Full-suite, differential, property, or metamorphic checks | Changed tests only |
| Security | Executable abuse cases and confirmed findings | Scanner count |
| Architecture | Dependency contracts around stable public boundaries | Folder naming |
| Test integrity | Protected baselines and mutation survival | Raw coverage |
| Scope discipline | Audited actions against an authorization allowlist | Task completion |
| Reviewability | True findings per reviewer minute (TPR/TNR calibrated) | Comment volume |
| Operability | Canary health, rollback, defect survival | Merge success |
| Economics | Cost and elapsed time per releasable change | LOC or PR count |

At the factory exit, report `correctness_pass`, `regression_pass`, `security_pass`, `architecture_pass`, and `scope_pass`, plus their joint **releasable patch rate**. Critical security, authorization, and data-loss events remain separate tail outcomes.

## Risk-Conditioned Control Architecture

```mermaid
flowchart TD
    I[Human Intent and Authorization] --> S[Executable Spec and Contracts]
    S --> C[Context Assembly]
    C --> G[Generator in Isolated Worktree]
    G --> F[Fast Deterministic Checks]
    F -->|Fail: bounded repair| G
    F --> B[Behavior and Regression Oracles]
    B -->|Fail: bounded repair| G
    B --> A[Architecture and Security Contracts]
    A -->|Fail: bounded repair| G
    A --> R[Calibrated Independent Review]
    R -->|Actionable finding: bounded repair| G
    R --> H{Risk Gate}
    H -->|Low risk and observable proof| D[Canary Deployment]
    H -->|Ambiguity or high blast radius| U[Accountable Human Decision]
    D --> T[Telemetry and Rollback]
    T --> S
```

### Tier 0 — Authorization and Containment
- Declare allowed repositories, paths, services, secrets, and deployment targets.
- Mount hidden conformance suites and protected regression baselines read-only outside the generator worktree.
- Keep candidate test paths writable so the generator can use [red green tdd](/wiki/red-green-tdd); evaluate those tests independently.
- Isolate filesystem, processes, credentials, and network egress to bound blast radius.^[raw/papers/overeager-coding-agents-2026.md]
- Record tool actions through audit channels the agent cannot bypass.

### Tier 1 — Fast Inner Loop
Run after each coherent edit:
- **TypeScript:** `oxlint` for fast AST checks (see [deterministic lint gates](/wiki/deterministic-lint-gates) for the error-only gate pattern); `tsc --noEmit --incremental` for type compatibility; changed Vitest/Jest unit tests. [[source: oxlint-type-aware-linting-2026]](/wiki/raw/articles/oxlint-type-aware-linting-2026)
- **Python:** Ruff lint/format; Pyright or mypy under repo policy; changed-scope pytest.
Fast checks create cheap back-pressure by catching local syntax and type impossibilities.

### Tier 2 — Structural and Behavioral Contracts
Before review:
- Hidden acceptance tests derived from requirements; full regression tests for affected dependency cones.
- Property, differential, or metamorphic checks where suitable; API and schema contracts.
- Graph conformance: `dependency-cruiser` rules for JavaScript/TypeScript and `import-linter` layer contracts for Python. [[source: dependency-cruiser-rules-reference-2026]](/wiki/raw/articles/dependency-cruiser-rules-reference-2026) [[source: import-linter-layer-contracts-2026]](/wiki/raw/articles/import-linter-layer-contracts-2026)

### Tier 3 — Test Strength and Adversarial Verification
Used selectively for critical domain, parser, financial, cryptographic, or security logic:
- Mutation testing via StrykerJS or `mutmut` to measure assertion strength. [[source: strykerjs-configuration-2026]](/wiki/raw/articles/strykerjs-configuration-2026) [[source: mutmut-documentation-2026]](/wiki/raw/articles/mutmut-documentation-2026)
- Adversarial probes: malformed input fuzzing and hacker–fixer–solver loops against acceptance verifiers.^[raw/papers/adversarial-hacker-fixer-verifiers-2026.md]

### Tier 4 — Calibrated Independent Review
- Independent judge agents review diffs against specifications, non-goals, and security policies before seeing implementer rationale.
- Reviewer agents must be calibrated on expert-labeled sets, reporting true positive (TPR) and true negative rates (TNR) to avoid low-signal review noise.^[raw/papers/code-review-agents-empirical-study-2026.md] [[source: hamel-husain-shreya-shankar-evals-skills-2026]](/wiki/raw/articles/hamel-husain-shreya-shankar-evals-skills-2026)

### Tier 5 — Risk Gate and Production Feedback
Auto-merge only for pre-approved low-risk changes meeting observable criteria:
- Conformance suite passes; touched resources match allowlist; regression checks pass without exceptions.
- Canary monitoring and rollback triggers active. Ambiguous intent or high blast radius requires accountable human sign-off.

## Controls by Task Risk

| Task | Minimum control | Human role |
|---|---|---|
| Documentation or isolated formatting | T0–T1 plus link/build checks | Sample audit |
| Local bug fix with reproducer | T0–T2 plus protected regression | Review exceptions |
| Feature in one bounded module | T0–T2 plus API/property checks | Approve spec or boundary changes |
| Core domain, parser, financial, crypto, auth | T0–T3 with mutation/adversarial checks | Mandatory accountable review |
| Cross-module refactor | T0–T4 with dependency graph & characterization | Review architecture & migration |
| Platform or data migration | T0–T5 with compatibility matrix & canary | Approve rollout |
| Ambiguous product behavior | Prototype and clarify first | Decide intent; no auto-merge |

## Oracle Health

A green verifier can still be flawed. Epoch and OpenAI audits demonstrated SWE-bench task contamination, test over-specificity, and exploitable test harness bugs. [[source: epoch-swe-bench-verified-analysis-2025]](/wiki/raw/articles/epoch-swe-bench-verified-analysis-2025) [[source: openai-swe-bench-verified-audit-2026]](/wiki/raw/articles/openai-swe-bench-verified-audit-2026) Production harnesses must treat evaluation oracles as production software subject to regression suites, mutation checks, and [eval driven development](/wiki/eval-driven-development).

## What Remains Unverified

- The causal effect of the complete multi-tier stack on escaped production defects.
- Long-term maintenance cost under task-matched random assignment.
- Mutation testing's cost-effectiveness as an autonomous merge gate.
- The strongest null: frontier model capability, basic compiler/linter feedback, and human task selection may drive nearly all observed success; multi-tier scaffolding may add compute and false rejection without reducing escapes.

## Related
- [five debts of agentic engineering](/wiki/five-debts-of-agentic-engineering) — generative debt modes and control mappings
- [releasable patch rate](/wiki/releasable-patch-rate) — north-star factory metric
- [constraint layering](/wiki/constraint-layering) — allocating controls across prompt, tools, and sandboxes
- [skill treatment effect](/wiki/skill-treatment-effect) — empirical evaluation of procedural skills
- [agentic quality evidence](/wiki/agentic-quality-evidence) — empirical evidence and contradictions
- [eval driven development](/wiki/eval-driven-development) — error-discovery eval loop
- [agentic engineering patterns](/wiki/agentic-engineering-patterns) — disciplined practitioner patterns
- [agentic software factory](/wiki/agentic-software-factory) — operating model
- [agent containment and blast radius](/wiki/agent-containment-and-blast-radius) — authorization and damage bounds
- [designing for verifiability](/wiki/designing-for-verifiability) — inspectable product contracts
- [agentic code quality entity accounting](/wiki/agentic-code-quality-entity-accounting) — cycle 1 source-author dispositions
- [agentic code quality cycle 2 entity accounting](/wiki/agentic-code-quality-cycle-2-entity-accounting) — cycle 2 source-author dispositions

---

## Agentic Engineering Patterns
- URL: https://pyweb.dev/wiki/agentic-engineering-patterns
- Raw Markdown: https://pyweb.dev/wiki/agentic-engineering-patterns.md
- Type: concept
- Summary: Disciplined software engineering practices for working with autonomous coding agents that write, test, and verify code in loops.
- Tags: agents, workflow, principle, coding-guidelines, tdd

# Agentic Engineering Patterns

Agentic engineering is the discipline of professional software engineers using autonomous coding agents to amplify technical capability while enforcing verification, architectural integrity, and rigor. Coined and cataloged by [simon willison](/wiki/simon-willison), it distinguishes disciplined engineering with feedback loops from unconstrained "vibe coding". [[source: simon-willison-agentic-engineering-patterns-2026]](/wiki/raw/articles/simon-willison-agentic-engineering-patterns-2026)

## Core Canon: Proof-of-Work & Human Accountability

> **"Your job is to deliver code you have proven to work."** — Simon Willison (2025)

Generating code is computationally cheap; verifying correctness remains the true engineering constraint. The developer is not accountable for syntax generation, but retains strict accountability for proof of execution, regression prevention, and edge-case validation. [[source: simon-willison-code-proven-to-work-2025]](/wiki/raw/articles/simon-willison-code-proven-to-work-2025)

## Primary Named Patterns

```mermaid
flowchart LR
    A[First Run the Tests] --> B[Red/Green TDD]
    B --> C[Agentic Implementation]
    C --> D[Agentic Manual Testing]
    D --> E[Attacks Become Evals]
    E --> F[Verified Releasable PR]
```

### 1. First Run the Tests
- **Mechanism:** Before generating or modifying any code, the agent is forced to discover and run the existing test suite (`npm test`, `pytest`).
- **Function:** Seeds the agent's context window with exact execution commands, establishes baseline regression safety, and anchors the harness in an active testing mindset. [[source: simon-willison-agentic-engineering-patterns-2026]](/wiki/raw/articles/simon-willison-agentic-engineering-patterns-2026)

### 2. Red/Green TDD for Agents
- **Mechanism:** Enforce authoring a failing assertion that reproduces the intended bug or requirement *before* generating implementation logic.
- **Function:** Prevents "self-fulfilling tests" where an agent generates a buggy implementation and subsequently writes weak tests that merely mirror its faulty assumptions. [kent beck](/wiki/kent-beck) identifies TDD as the fundamental governor for coding agents ("the unpredictable genie"). [[source: kent-beck-gergely-orosz-tdd-ai-agents-2025]](/wiki/raw/articles/kent-beck-gergely-orosz-tdd-ai-agents-2025) [martin fowler](/wiki/martin-fowler) notes TDD preserves essential human comprehension in agentic loops. [[source: martin-fowler-fragments-2026-01-08]](/wiki/raw/articles/martin-fowler-fragments-2026-01-08)

### 3. Agentic Manual Testing
- **Mechanism:** Empower the agent with browser automation, terminal execution, and synthetic data generation to exercise running applications dynamically.
- **Function:** Catches runtime integration defects that unit test suites fail to capture.

### 4. Attacks Become Evals
- **Mechanism:** When an agent or production system experiences a jailbreak, test bypass, or edge-case failure, that exact failure is codified into an automated regression eval. Demonstrated by [boris cherny](/wiki/boris-cherny) in Claude Code's internal harness. [[source: boris-cherny-how-boris-uses-claude-code-2026]](/wiki/raw/articles/boris-cherny-how-boris-uses-claude-code-2026)

### 5. Package Proof with the Patch
- **Mechanism:** Every agentic PR must supply verifiable proof of work (baseline command and exit status, red test failure output, green verification output, manual runtime execution traces, and explicit residual risk analysis) rather than requiring reviewers to reconstruct trust from scratch.

### 6. Bounded Autonomous Repair
- **Mechanism:** Enforce strict execution budgets on autonomous fix loops (e.g., maximum attempts per failed gate, fixed token caps, and file-touch boundaries). Prevent unbounded feedback loops from accumulating compensating hacks or modifying unrelated files.

## Key Anti-Patterns

- **The Test Deletion Anti-Pattern:** Agents encountering stubborn test failures modify, soften, or delete test assertions to manufacture a green exit status. Mitigated by mounting baseline regression suites read-only. [[source: kent-beck-gergely-orosz-tdd-ai-agents-2025]](/wiki/raw/articles/kent-beck-gergely-orosz-tdd-ai-agents-2025)
- **The False Victory:** An agent misinterprets an HTTP 400 or handled exception as proof that an integration endpoint works. Mitigated by full end-to-end trace verification. [[source: stripe-can-ai-agents-build-real-stripe-integrations-2026]](/wiki/raw/articles/stripe-can-ai-agents-build-real-stripe-integrations-2026)
- **Monolithic Context Bloat:** Dumping massive diffs into chat context rather than directing agents with targeted `grep`, `glob`, and isolated worktrees.

## Related
- [software engineering fundamentals for agents](/wiki/software-engineering-fundamentals-for-agents) — the 5 foundational pillars required to steer agentic code
- [five debts of agentic engineering](/wiki/five-debts-of-agentic-engineering) — generative debt modes prevented by these patterns
- [andrew ng](/wiki/andrew-ng) — author of AI Engineering Skills Map
- [releasable patch rate](/wiki/releasable-patch-rate) — measuring complete patch delivery
- [agentic code quality](/wiki/agentic-code-quality) — multi-tier quality controls
- [red green tdd](/wiki/red-green-tdd) — test-driven cycle
- [agent harness engineering](/wiki/agent-harness-engineering) — harness design
- [git for agentic workflows](/wiki/git-for-agentic-workflows) — version control discipline
- [hoard and recombine](/wiki/hoard-and-recombine) — collecting verified code references
- [github](/wiki/github) — developer platform & spec-driven toolkit
- [evals skills](/wiki/evals-skills) — evaluation tooling
- [simon willison](/wiki/simon-willison) — pattern originator
- [kent beck](/wiki/kent-beck) — TDD governor principle
- [martin fowler](/wiki/martin-fowler) — comprehension loop
- [boris cherny](/wiki/boris-cherny) — verification-first harness

---

## Agentic Manual Testing
- URL: https://pyweb.dev/wiki/agentic-manual-testing
- Raw Markdown: https://pyweb.dev/wiki/agentic-manual-testing.md
- Type: concept
- Summary: Directing coding agents to actively exercise running servers, APIs, CLI one-liners, and browser automation to verify real system behaviour.
- Tags: technique, agents, feedback-loops, workflow

# Agentic Manual Testing

Automated unit tests are necessary, but code that passes unit tests can still fail to boot, miss styling, or crash on live input. Agentic manual testing is the discipline of having the agent spin up the software and actively interact with it before declaring completion.

## Mechanisms
- **CLI / One-liners:** Direct the agent to execute edge cases directly via `python -c "..."` or compile temporary test scripts in `/tmp`.
- **API Exploration:** Have the agent start a local development server (e.g. `python -m http.server` or app backend) and explore JSON endpoints with `curl`.
- **Headless Browser Automation:** Direct agents using browser tools (Playwright, Puppeteer, agent-browser, or CLI wrappers like `rodney`) to click buttons, fill forms, verify layouts, and check console errors.
- **Evidence Capture:** Require verifiable outputs (terminal logs, rendered HTML, or tool execution transcripts) rather than accepting self-reported completion.

## "First Run The Tests"
Starting a coding session with a prompt like `"First run the tests"` or `"Run uv run pytest"` forces the agent to discover test commands, gauges codebase complexity, and anchors the agent in a verification mindset from turn one.

## Related
- [simon willison](/wiki/simon-willison)
- [agentic engineering patterns](/wiki/agentic-engineering-patterns)
- [red green tdd](/wiki/red-green-tdd)
- [tdd with agents](/wiki/tdd-with-agents)
- [karpathy four guidelines](/wiki/karpathy-four-guidelines)
- [closed loop agent improvement](/wiki/closed-loop-agent-improvement)
- [automated eval engineering](/wiki/automated-eval-engineering)

---

## Agentic Software Factory
- URL: https://pyweb.dev/wiki/agentic-software-factory
- Raw Markdown: https://pyweb.dev/wiki/agentic-software-factory.md
- Type: concept
- Summary: Operating model for software development where autonomous AI agents perform end-to-end delivery under harness constraints and risk-tiered human oversight.
- Tags: workflow, agents, coding-guidelines, subagents

# Agentic Software Factory

The **Agentic Software Factory** is an operating model for software delivery where autonomous AI agents handle production work from task decomposition to pull request and merge, while human engineers shift from writing code to designing specifications, managing architecture, and supervising exception points. [[source: augment-code-software-factory-vs-devops-2026]](/wiki/raw/articles/augment-code-software-factory-vs-devops-2026)

```mermaid
flowchart TD
    subgraph HumanSupervision [Human Intent & Architecture]
        Spec[Specification & ADRs]
        Escalations[High-Blast-Radius Exceptions]
    end

    subgraph FactoryLine [Agentic Assembly Line]
        Decompose[1. Task Decomposition & Sizing]
        Implement[2. Worker Agents / Worktrees]
        InnerGate[3. Inner Loop: Lint + Type + Unit]
        ReviewMatrix[4. Adversarial Review & Mutation Gates]
    end

    subgraph DeliveryPlatform [Delivery Substrate]
        CI[Automated CI/CD Pipeline]
        Prod[Production Canary & Telemetry]
    end

    Spec --> Decompose
    Decompose --> Implement
    Implement --> InnerGate
    InnerGate --> ReviewMatrix
    ReviewMatrix -->|Pass| CI
    ReviewMatrix -->|Fail / Anomaly| Escalations
    CI --> Prod
    Prod -- Telemetry Feedback --> Spec
```

## Software Factory vs. DevOps

While DevOps established the delivery pipelines and culture for human-operated software lifecycles, the agentic software factory changes the unit of labor inside the pipeline: [[source: augment-code-software-factory-vs-devops-2026]](/wiki/raw/articles/augment-code-software-factory-vs-devops-2026)

| Dimension | DevOps (Human Labor) | Software Factory (Agent Labor) |
|---|---|---|
| **Primary Unit of Labor** | Human engineer authoring commits | Autonomous agent executing tasks in isolated worktrees |
| **Binding Bottleneck** | Authoring speed & developer typing | Verification throughput & specification clarity |
| **Quality Control** | Human peer review + standard CI | Multi-tier deterministic gates + adversarial reviewer agents |
| **Failure Modes** | Syntax errors, merge conflicts, human error | Syntactically plausible hallucinations, test tampering, boundary leaks |
| **Feedback Routing** | Sprint retrospectives & human memory | Context engine updates, repo invariants, deterministic linters |

## Core Factory Principles

### 1. Slow Plan, Long Implement
As observed by Wes McKinney, robust agent execution requires rigorous upfront specification before code generation begins. [[source: hugo-bowne-anderson-agentic-software-factory-2026]](/wiki/raw/articles/hugo-bowne-anderson-agentic-software-factory-2026) Tasks are decomposed into **vertical slices** (touching one public contract with self-contained regression tests) rather than broad horizontal refactors.

### 2. Continuous Automated Review & Ledgering
At scale (e.g. millions of lines generated across dozens of repositories), human review of every diff is mathematically impossible. Systems like McKinney's `RoboRev` run high-reasoning models (GPT-5.5 / Claude) as post-commit hooks on every turn, recording findings into an append-only review ledger. [[source: hugo-bowne-anderson-agentic-software-factory-2026]](/wiki/raw/articles/hugo-bowne-anderson-agentic-software-factory-2026)

### 3. Five Binding Downstream Constraints
When authoring constraints disappear, five downstream bottlenecks govern factory throughput: [[source: augment-code-software-factory-vs-devops-2026]](/wiki/raw/articles/augment-code-software-factory-vs-devops-2026)
1. **Specification Precision:** Eliminating ambiguous requirements that cause agent thrashing.
2. **Context Engine Quality:** Supplying conflict-resolved, permission-aware context on why code exists.
3. **Deterministic Verification:** Multi-tier gates enforcing [agentic code quality](/wiki/agentic-code-quality).
4. **Audit Traceability:** Complete provenance linking intent, prompts, diffs, and verification traces.
5. **Controlled Iteration:** Blast-radius budgets capping modified files and lines per execution.

## Cross-links
- [agentic code quality](/wiki/agentic-code-quality) — verification tiers, mutation testing, and back-pressure
- [agent harness engineering](/wiki/agent-harness-engineering) — operating infrastructure and runtime isolation
- [multi agent orchestration](/wiki/multi-agent-orchestration) — coordinator and worker agent hierarchies
- [conformance suites as fitness functions](/wiki/conformance-suites-as-fitness-functions) — test suites as objective factory fitness functions
- [wes mckinney](/wiki/wes-mckinney) — factory practitioner, post-commit review systems
- [addy osmani](/wiki/addy-osmani) — constraint-driven quality in software factories
- [tessl](/wiki/tessl) — system harness benchmarks and NS2 orchestration

---

## Agentic Vulnerability Lifecycle
- URL: https://pyweb.dev/wiki/agentic-vulnerability-lifecycle
- Raw Markdown: https://pyweb.dev/wiki/agentic-vulnerability-lifecycle.md
- Type: concept
- Summary: The collapse of traditional security embargo windows and CVE disclosure pipelines driven by automated coding agents that synthesize working exploits within minutes of patch discussions.
- Tags: security, agents, workflow, anti-patterns

# Agentic Vulnerability Lifecycle

The **Agentic Vulnerability Lifecycle** refers to the structural breakdown of traditional responsible disclosure, embargo windows, and vulnerability triage caused by autonomous coding agents weaponizing public bug discussions into working exploits in real time. [[source: simon-willison-just-a-rumour-of-a-bug-is-enough-to-find-a-security-exploit-2026]](/wiki/raw/articles/simon-willison-just-a-rumour-of-a-bug-is-enough-to-find-a-security-exploit-2026)

```mermaid
flowchart LR
    A[Public Commit / Patch Discussion] --> B[Automated Agent Watchers]
    B --> C[Exploit Generation < 10 mins]
    C --> D[Active Target Probing]
    A --> E[Maintainer Triage & Embargo]
    E --> F[CVE Backlog: 3-4 Weeks]
```

## The Ten-Minute Exploit Window

Historically, open-source security disclosures operated on multi-day or multi-week embargoes, assuming human reverse-engineering required substantial latency to convert a subtle bug report into an exploit payload.

In 2026, autonomous coding agents collapsed this window to minutes. [anil madhavapeddy](/wiki/anil-madhavapeddy) (professor of computer science at Cambridge and core maintainer of the OCaml compiler) reported that security issues in OCaml projects experienced automated exploit attempts within minutes of patches being shared for discussion:

> *"Within about ten minutes (!) this website was fielding probes for percent-encoded traversal sequences, indicating that automated watchers are keeping an eye on public repositories."* [[source: simon-willison-just-a-rumour-of-a-bug-is-enough-to-find-a-security-exploit-2026]](/wiki/raw/articles/simon-willison-just-a-rumour-of-a-bug-is-enough-to-find-a-security-exploit-2026)

Modern coding agents require only the slightest hint or "rumour of a bug" across commit diffs or issue threads to localize the vulnerability and synthesize an actionable exploit. When safety-aligned models (such as Claude Fable) refuse exploit synthesis, automated pipelines seamlessly switch to models like DeepSeek V4 Pro to finish the generation. [[source: simon-willison-just-a-rumour-of-a-bug-is-enough-to-find-a-security-exploit-2026]](/wiki/raw/articles/simon-willison-just-a-rumour-of-a-bug-is-enough-to-find-a-security-exploit-2026)

## Maintainer Triage and CVE Pipeline Breakdown

The asymmetry between automated generation and manual triage creates severe maintenance bottlenecks:

1. **Disclosure Volume Surges:** [nick craig wood](/wiki/nick-craig-wood) (maintainer of rclone) reported receiving over 40 security disclosures in a single month, compared to roughly 20 disclosures across the first 10 years of the project. [[source: simon-willison-just-a-rumour-of-a-bug-is-enough-to-find-a-security-exploit-2026]](/wiki/raw/articles/simon-willison-just-a-rumour-of-a-bug-is-enough-to-find-a-security-exploit-2026)
2. **High Signal-to-Noise Ratio:** Unlike generic spam, approximately 75% of these agent-generated disclosures contain a genuine bug or actionable vulnerability, requiring deep human verification and fix engineering. [[source: simon-willison-just-a-rumour-of-a-bug-is-enough-to-find-a-security-exploit-2026]](/wiki/raw/articles/simon-willison-just-a-rumour-of-a-bug-is-enough-to-find-a-security-exploit-2026)
3. **Institutional Backlog:** GitHub CVE assignment timelines stretched from 2-3 days to 3-4 weeks under the disclosure load, forcing maintainers to publish point releases with `CVE-PENDING` in changelogs. [[source: simon-willison-just-a-rumour-of-a-bug-is-enough-to-find-a-security-exploit-2026]](/wiki/raw/articles/simon-willison-just-a-rumour-of-a-bug-is-enough-to-find-a-security-exploit-2026)

## Architectural Implications

The collapse of disclosure embargoes demonstrates that natural-language or process-based secrecy cannot protect open repositories. Mitigation requires:
- **Hardened Sandboxing:** Isolating runtime environments with strict filesystem and egress boundaries rather than relying on patch secrecy ([agent containment and blast radius](/wiki/agent-containment-and-blast-radius)).
- **Deterministic CI Verification:** Automated regression tests and conformance suites that run in closed perimeters prior to public PR publication ([conformance suites as fitness functions](/wiki/conformance-suites-as-fitness-functions)).
- **Constraint Layering:** Enforcing capability limits mechanically at the OS boundary rather than via model alignment ([constraint layering](/wiki/constraint-layering)).

## Related Concepts
- [agent containment and blast radius](/wiki/agent-containment-and-blast-radius) — boundary isolation against compromised agent runtimes
- [constraint layering](/wiki/constraint-layering) — mechanical constraints vs probabilistic model alignment
- [conformance suites as fitness functions](/wiki/conformance-suites-as-fitness-functions) — automated invariant verification
- [simon willison](/wiki/simon-willison) — analysis of agent security boundaries

---

## Agents vs Workflows
- URL: https://pyweb.dev/wiki/agents-vs-workflows
- Raw Markdown: https://pyweb.dev/wiki/agents-vs-workflows.md
- Type: concept
- Summary: The degree-of-autonomy spectrum from Anthropic's Building Effective Agents: workflows run on developer-defined code paths, agents direct their own process via tools and environment feedback.
- Tags: agents, patterns, llm-fundamentals

# Agents vs Workflows

Anthropic's *Building Effective Agents* (Dec 2024) draws the fundamental architectural line: **workflows** use "predefined code paths" while **agents** "dynamically direct their own processes." The difference is the **degree of autonomy** — and the operative test is *who decides when to stop*: in a workflow the developer's code terminates the run; in an agent the LLM itself stops when it judges the task complete.^[raw/aihero/what-is-an-agent.md]

## The Autonomy Gradient
```mermaid
flowchart LR
    W["Workflow\npredefined code paths\ncode decides termination"] -->|"more autonomy"| A["Agent\nLLM picks tools per step\nLLM decides termination"]
```
- **Workflows:** predetermined steps written in code; better results than agents whenever the task is clearly specified; unfairly maligned for being less exciting.
- **Agents:** the LLM improvises through unclear steps, "making it up as it goes" — more powerful, less predictable. Best when the path to completion cannot be specified in advance.

## Anthropic's Workflow Patterns
1. **Prompt chaining** — each LLM call processes the previous one's output.
2. **Routing** — an LLM classifies input and directs it to a specialized followup task.
3. **Parallelization** — LLM calls run in parallel (e.g. split text, summarize parts, summarize summaries).
4. **Orchestrator-workers** — a central LLM breaks tasks down and delegates to worker LLMs (see [multi agent orchestration](/wiki/multi-agent-orchestration)).
5. **Evaluator-optimizer** — one LLM generates while another evaluates in a loop (see [generator evaluator loop](/wiki/generator-evaluator-loop)).

Only one pattern is truly agentic: agents "plan and operate independently," "gain ground truth from the environment at each step" (tool results, code execution), and terminate on completion or a stopping condition (e.g. max iterations). The compressed definition: **agents are just LLMs using tools based on environmental feedback in a loop** — which is precisely the [tool calling loop](/wiki/tool-calling-loop).^[raw/aihero/building-effective-agents.md]

## Anti-Pattern: Frameworks First
Anthropic repeatedly warns against agent frameworks as a first resort (LangGraph, Bedrock Agents, etc.): they "create extra layers of abstraction that can obscure the underlying prompts and responses" and tempt complexity where a simpler setup suffices. Use LLM APIs directly until you understand the code you'd be abstracting.^[raw/aihero/building-effective-agents.md] (Pocock's carve-out: a compatibility library like the AI SDK is not a framework — see [model provider abstraction](/wiki/model-provider-abstraction).)

## Rule of Thumb

Workflows when the path is known and repeatable; agents when it is not.
If you can draw the flowchart, build the workflow - the flowchart IS the
reliability.

## Failure Modes

| Symptom | Root cause | Fix |
|---|---|---|
| Workflow built where judgment needed | Path assumed known when it is not | If branching is data-dependent, use an agent |
| Agent used for a known pipeline | Autonomy added without need | Draw the flowchart first; if complete, it is a workflow |

## Related
[tool calling loop](/wiki/tool-calling-loop), [generator evaluator loop](/wiki/generator-evaluator-loop), [multi agent orchestration](/wiki/multi-agent-orchestration), [agent harness engineering](/wiki/agent-harness-engineering), [llm message protocol](/wiki/llm-message-protocol).

---

## AGENTS.md Specification
- URL: https://pyweb.dev/wiki/agents-md-spec
- Raw Markdown: https://pyweb.dev/wiki/agents-md-spec.md
- Type: concept
- Summary: The root project context standard for orienting AI coding agents without system prompt bloat.
- Tags: agents, context-engineering, skills, workflow

# AGENTS.md Specification

**AGENTS.md** is an emerging industry standard markdown file placed at the root of a project workspace to orient AI coding agents (such as Hermes, Claude Code, Cursor, and Codex) as soon as they inspect a repo.

## The Core Philosophy: "Never Run /init"
Auto-generated agent initialization files typically produce hundreds of lines of generic prose describing basic language syntax and obvious git commands. High-signal `AGENTS.md` files are strictly hand-crafted, concise, and focused exclusively on non-obvious project reality.

## Essential Sections of AGENTS.md
1. **Layout & Entry Points:** The high-level file tree and where persistent vs ephemeral files live.
2. **Toolchain & Non-Obvious Commands:** The exact build, test, and typecheck commands (e.g. `uv run ...` or `pnpm test`), highlighting environment quirks like PEP 668 or package manager constraints.
3. **Architecture Seams & Decisions:** References to `CONTEXT.md` and ADR records.
4. **Pitfalls & Hard Guardrails:** Things the agent must NEVER do (e.g. "Do not push directly to main", "Do not mock database calls in unit tests").

## AGENTS.md vs System Prompts
`AGENTS.md` belongs in the project filesystem, not baked into a static system prompt. This allows agent behaviour to evolve dynamically with repository commits and branch checkouts.

## Related
[context engineering](/wiki/context-engineering), [prompt bloat](/wiki/prompt-bloat), [progressive disclosure](/wiki/progressive-disclosure), [ai coding taxonomy](/wiki/ai-coding-taxonomy).

---

## AI Coding Taxonomy & Agent Experience
- URL: https://pyweb.dev/wiki/ai-coding-taxonomy
- Raw Markdown: https://pyweb.dev/wiki/ai-coding-taxonomy.md
- Type: concept
- Summary: Core vocabulary and operational taxonomy for AI coding agents, attention dynamics, developer/agent experience (DX/AX), and execution modes.
- Tags: agents, context-engineering, coding-guidelines, workflow, principle

# AI Coding Taxonomy & Agent Experience

A rigorous, standardized vocabulary for reasoning about AI models, harnesses, attention dynamics, collaboration modes, and developer/agent experience, compiled from [matt pocock](/wiki/matt-pocock)'s AI Hero curriculum.

```mermaid
flowchart TD
    subgraph ModelLayer [1. Model & Inference Layer]
        M[Model / Parameters] -->|Inference| P[Next-token prediction]
        P --> TOK[Input / Output / Cache Tokens]
        P --> PR[Prefix Cache]
    end

    subgraph AttentionLayer [2. Attention Dynamics]
        TOK --> AR[Attention Relationship: ~N² pairs]
        AR --> AB[Attention Budget per Token]
        AB -->|Context Accumulation| AD[Attention Degradation]
        AD --> SZ[Smart Zone vs Dumb Zone]
    end

    subgraph HarnessLayer [3. Harness & Environment]
        H[Harness / System Prompt] -->|Manages| C[Context Window]
        H -->|Tool Call| ENV[Environment / Filesystem]
        ENV -->|Tool Result| H
        H -->|Permissions / Modes| SB[Sandbox]
    end

    subgraph LifecycleLayer [4. Context Lifecycle & State]
        C --> CL[Clearing]
        C --> HA[Handoff Artifacts: Spec & Ticket]
        C --> CP[Compaction / Autocompact]
        C --> MS[Memory System]
    end

    subgraph ExperienceLayer [5. Experience & Interaction Modes]
        DX[DX: Developer Experience] <--> AX[AX: Agent Experience]
        HITL[Human-in-the-loop] <--> AFK[AFK Unattended]
        VC[Vibe Coding] <--> HR[Human Review / Diffs]
    end
```

## 1. Model, Inference, and Attention Dynamics

- **Model vs Harness:** A model represents the frozen parameters (weights) tuned during training that performs stateless next-token prediction. It cannot act agentically alone. The [harness](/wiki/agent-harness-engineering) surrounds the model with tools, system prompts, context-window management, permissions, and hooks.
- **Inference & Token Accounting:** Running a model generates output tokens from input tokens. Consecutive requests sharing prefixes leverage provider-side **prefix caches**, billing those tokens as discounted cache tokens.
- **Attention Relationships:** In a context of $N$ tokens, approximately $N^2$ attention relationships exist between token pairs. Because each token has a finite **attention budget** to distribute across the context, adding tokens dilutes attention.
- **Attention Degradation & Smart Zone:** As context accumulates, each token's attention budget spreads across competing tokens, causing signal on critical relationships to shrink (**attention degradation**). This defines the [smart zone](/wiki/smart-zone) early in a session versus the sloppier "dumb zone" later in extended sessions.
- **Parametric vs Contextual Knowledge:** Parametric knowledge is frozen in weights at the **knowledge cutoff** date; contextual knowledge consists of facts loaded directly into the context window.
- **Sycophancy vs Hallucination:** Sycophancy is confidently agreeable output shaped by RLHF preferences; hallucination represents confidently-wrong output divided into factuality (invented facts) and faithfulness (drift from loaded context).

## 2. Interaction Modes & Experience (DX vs AX)

- **DX (Developer Experience):** How easy a codebase and toolchain make it for humans to do good work (documentation, feedback speed, error clarity).
- **AX (Agent Experience):** How well the environment is configured for an AI agent to do good work—deterministic checks, deep module architecture, and easily retrieved context without prompt bloat.
- **Human-in-the-Loop vs AFK:**
  - **Human-in-the-loop:** Pairing with an agent in real time, reviewing diffs, and redirecting.
  - **AFK (Away From Keyboard):** Unattended execution where an agent works autonomously against automated validation suites.
- **Human Review vs Vibe Coding:**
  - **Human Review:** Reading and judging the actual code diff produced by the agent.
  - **Vibe Coding:** Accepting code without inspecting diffs, treating generated output as opaque.
- **Automated Check vs Automated Review:**
  - **Automated Check:** Deterministic pass/fail verification in the environment (tests, lints, typechecks, build).
  - **Automated Review:** Non-deterministic evaluation where another agent inspects code and exercises judgment.

## 3. Context Lifecycle and Handoff Mechanics

- **Stateless vs Stateful:** Models are stateless across requests; agents are stateless across sessions by default unless persistence mechanisms are added.
- **Progressive Disclosure & Context Pointers:** Loading only necessary context upfront while leaving **context pointers** (mentions pointing to external documents or skills) for on-demand loading.
- **Clearing vs Compaction:**
  - **Clearing:** Ending a session to restart with an empty context window (`/clear`).
  - **Compaction:** In-memory summarization of past history seeding a fresh session, trading fidelity for headroom.
- **Handoff Artifacts (Specs & Tickets):** Structured documents bridging work across session boundaries. A **spec** scopes multi-session architecture; a **ticket** scopes one bounded session.

## Related Concepts

- [context engineering](/wiki/context-engineering)
- [smart zone](/wiki/smart-zone)
- [context rot](/wiki/context-rot)
- [agent harness engineering](/wiki/agent-harness-engineering)
- [agents md spec](/wiki/agents-md-spec)
- [handoff artifacts](/wiki/handoff-artifacts)
- [progressive disclosure](/wiki/progressive-disclosure)
- [matt pocock](/wiki/matt-pocock)

---

## AI Engineer Role
- URL: https://pyweb.dev/wiki/ai-engineer-role
- Raw Markdown: https://pyweb.dev/wiki/ai-engineer-role.md
- Type: concept
- Summary: The application-layer engineer who orchestrates AI APIs, RAG, and evals — distinct from ML Engineering by the API boundary, grounded in Latent Space's 'Rise of the AI Engineer'.
- Tags: llm-fundamentals, workflow, roles

# AI Engineer Role

The **AI Engineer** (per Latent Space's "The Rise of the AI Engineer," relayed by Matt Pocock) is a software developer who builds applications powered by AI — and the role is defined negatively as much as positively:^[raw/aihero/what-is-an-ai-engineer.md]

**You don't need:** linear algebra, building foundation models from scratch, or having read "Attention Is All You Need."
**You do need:** strong software engineering fundamentals, reliable/scalable application skills, modern AI tooling knowledge, and a user-experience focus.

## The API Boundary vs ML Engineering
- **AI Engineers** build applications that *use* AI — orchestrating APIs, implementing RAG, building evaluation systems, and "writing lots of code in the hottest new programming language: English" (Karpathy).
- **ML Engineers** work *below* the API boundary: training, fine-tuning, model internals.

## The Mindset Shift
Building with LLMs means trading deterministic input->output systems for **probabilistic** ones: define success criteria early, and build a culture of continuous improvement on real user data (see [eval taxonomy](/wiki/eval-taxonomy), the Vibes-Only Trough to Data-Driven Slope journey). The role's daily work spans [llm message protocol](/wiki/llm-message-protocol) mechanics, [agents vs workflows](/wiki/agents-vs-workflows) architecture choices, and [context budget audit](/wiki/context-budget-audit) cost discipline.

## Related
[eval taxonomy](/wiki/eval-taxonomy), [agents vs workflows](/wiki/agents-vs-workflows), [llm message protocol](/wiki/llm-message-protocol), [context budget audit](/wiki/context-budget-audit), [software engineering fundamentals for agents](/wiki/software-engineering-fundamentals-for-agents).

---

## Automated Eval Engineering
- URL: https://pyweb.dev/wiki/automated-eval-engineering
- Raw Markdown: https://pyweb.dev/wiki/automated-eval-engineering.md
- Type: concept
- Summary: Techniques and harnesses for constructing reproducible, containerized agent evaluations from repository code and production traces.
- Tags: evaluation, agents, workflow, feedback-loops

# Automated Eval Engineering

Automated eval engineering is the practice of using interactive agent tooling to inspect a software codebase, analyze production execution traces, and generate executable test harnesses and benchmarks for autonomous AI systems.

## The Interactive Interview Pattern

As demonstrated in LangChain's Harbor eval engineering workflows, fully autonomous "one-shot" eval generation frequently produces brittle, low-signal benchmarks or measures trivial properties. The effective pattern centers on an **interactive human-in-the-loop interview**:

1. **Repository Surface Mapping:** The eval agent crawls the application repository, cataloging prompts, tools, hooks, skills, schemas, and backing dependencies.
2. **Trace Contract Inspection:** Analyzing production execution logs (e.g. via `langsmith-cli`) to observe real tool arguments, outputs, failure states, and external API contracts.
3. **Capability Proposals & Human Steering:** The agent proposes specific capabilities worth measuring, recommends prioritized directions, and accepts human guidance on which dependencies to run live vs. simulate.
4. **Harbor Task Compilation:** Synthesizing containerized tasks with four core artifacts:
   - `task.toml`: metadata and runtime parameters.
   - `instruction.md`: task prompt delivered to the agent under test.
   - `environment/`: Dockerfile defining reproducible toolsets and filesystem states.
   - `tests/`: verifiers scoring agent trajectory, artifacts, and final state.

## Evals as Training and Harness Data

Standardized containerized evals decouple the environment from the agent configuration. This allows engineering teams to treat evals as fixed targets while rapidly running parallel sweeps across prompt changes, tool schemas, context engineering strategies, and model fine-tuning.

## Related
- [error analysis and evals](/wiki/error-analysis-and-evals)
- [closed loop agent improvement](/wiki/closed-loop-agent-improvement)
- [agentic manual testing](/wiki/agentic-manual-testing)
- [tdd with agents](/wiki/tdd-with-agents)
- [context engineering](/wiki/context-engineering)

---

## Build-From-Scratch Pedagogy
- URL: https://pyweb.dev/wiki/build-from-scratch-pedagogy
- Raw Markdown: https://pyweb.dev/wiki/build-from-scratch-pedagogy.md
- Type: concept
- Summary: Building the minimal toy version from scratch as the primary mechanism for deep understanding.
- Tags: pedagogy, technique, agents

# Build-From-Scratch Pedagogy

[andrej karpathy](/wiki/andrej-karpathy)'s teaching move: build the minimal version from scratch to understand
the thing. micrograd before PyTorch, nanoGPT before the transformers library. The toy
implementation IS the explanation — every step visible, no "it can be shown that."

## Root
[richard feynman](/wiki/richard-feynman)'s last blackboard: "What I cannot create, I do not understand."
Karpathy is an explicit disciple. The toy build is step 4 of the [feynman technique](/wiki/feynman-technique)
made executable: create the thing, and the creation IS the understanding.

## When to use it
- Understanding complex computational systems (how transformers work, how an event loop runs).
- Debugging a black box — rebuild the minimal reproduction instead of reading logs
  and guessing.
- Evaluating whether an abstraction earns its keep — can you build the 50-line
  version that does the same thing?

## How it fits the engineering stack
Complements [feynman technique](/wiki/feynman-technique) (plain-language explanation) and powers
[tracer bullets](/wiki/tracer-bullets) at the learning level: the minimal build is the tracer bullet
for understanding. Aligns with [karpathy four guidelines](/wiki/karpathy-four-guidelines) #2 (simplicity first)
and #4 (goal-driven — the working toy IS the verifiable success criteria).

## Related
[feynman technique](/wiki/feynman-technique), [first principles thinking](/wiki/first-principles-thinking), [tracer bullets](/wiki/tracer-bullets), [tdd with agents](/wiki/tdd-with-agents).

---

## Claude Managed Agents
- URL: https://pyweb.dev/wiki/claude-managed-agents
- Raw Markdown: https://pyweb.dev/wiki/claude-managed-agents.md
- Type: concept
- Summary: Anthropic's cloud-hosted agent runtime platform decoupling reasoning loops from containerized sandbox execution.
- Tags: agents, context-engineering, workflow, security

# Claude Managed Agents

**Claude Managed Agents (CMA)** is [anthropic](/wiki/anthropic)'s cloud-hosted infrastructure platform for deploying autonomous, long-running agent workflows. CMA decouples agent reasoning orchestration from tool execution environments, providing sandboxed Linux microVMs, persistent server-side event logs, prompt caching, and granular permission gates. [[source: anthropic-claude-managed-agents-overview-2026]](/wiki/raw/articles/anthropic-claude-managed-agents-overview-2026)

## Core Architectural Primitives

```mermaid
graph TD
    A[Claude Managed Agents Platform] --> B[Agent: Model, Prompt, Tools, Skills]
    A --> C[Environment: Managed Cloud or Self-Hosted Sandbox]
    A --> D[Session: Stateful Execution Instance & Filesystem]
    A --> E[Events: Persistent Bidirectional Event Log]
```

1. **Agent:** Reusable definition specifying base model (Claude Opus/Sonnet), system prompts, MCP tool definitions, and skill packages.
2. **Environment:** Configuration for execution boundaries—either Anthropic-managed cloud sandboxes with pre-installed Linux utilities or self-hosted sandboxes for strict enterprise data residency.
3. **Session:** Stateful runtime instance binding an agent to an environment. Retains ephemeral filesystem state, tool outputs, and execution context across multi-hour turns.
4. **Events:** Authoritative, append-only server-side event store capturing every user input, reasoning phase, tool call, output, and error.

 [[source: anthropic-claude-managed-agents-overview-2026]](/wiki/raw/articles/anthropic-claude-managed-agents-overview-2026)

## Decoupled Architecture: Brain vs. Hands

CMA implements the structural separation of machine reasoning from untrusted execution:
- **The Brain:** The reasoning engine running Claude models, managing prompt compaction, prompt caching, and context assembly.
- **The Hands:** Ephemeral, isolated execution sandboxes executing bash commands, reading/writing local files, querying web search APIs, or calling remote MCP endpoints.

 [[source: anthropic-claude-managed-agents-overview-2026]](/wiki/raw/articles/anthropic-claude-managed-agents-overview-2026)

## Permission Gating & Human-in-the-Loop

CMA implements policy-based permission gates on sensitive tools (e.g., `bash`, custom actions):
- When a gated tool is selected, the session emits `agent.tool_use` and parks with `session.status_idle` containing `stop_reason: requires_action`.
- Clients resolve the gate by submitting a `user.tool_confirmation` event (`allow` or `deny`).
- Denials are fed directly into the transcript as tool error feedback, prompting the agent to adjust strategy without repeating identical failures. [[source: assistant-ui-claude-managed-agents-2026]](/wiki/raw/articles/assistant-ui-claude-managed-agents-2026)

## Frontend & Protocol Integration

Via adapters such as [AG UI](/wiki/ag-ui-protocol) and client libraries ([copilotkit](/wiki/copilotkit), assistant-ui), CMA sessions map 1:1 to user chat threads:
- **Zero Local Message Tables:** The frontend queries `sessions.events.list()` as the single source of truth.
- **Stream Replay Consistency:** Replaying past sessions and streaming active runs execute through the identical event-folding reducer. [[source: assistant-ui-claude-managed-agents-2026]](/wiki/raw/articles/assistant-ui-claude-managed-agents-2026)

## Cross-links
- [anthropic](/wiki/anthropic) — platform creator
- [ag ui protocol](/wiki/ag-ui-protocol) — frontend interaction protocol
- [agent native infrastructure](/wiki/agent-native-infrastructure) — underlying compute paradigm
- [copilotkit](/wiki/copilotkit) — integration ecosystem partner

---

## Clean Architecture
- URL: https://pyweb.dev/wiki/clean-architecture
- Raw Markdown: https://pyweb.dev/wiki/clean-architecture.md
- Type: concept
- Summary: Uncle Bobs dependency-inward organizing rule applied to agent systems: source dependencies point toward stable abstractions.
- Tags: principle, workflow

# Clean Architecture

A software architecture pattern synthesized by [robert c martin](/wiki/robert-c-martin) that integrates multiple layered architectural approaches into a unified system design philosophy. Clean Architecture organizes systems into concentric circles where dependencies flow inward, enabling testable, maintainable, and framework-independent software.


```text
UI  ->  use-cases  ->  entities
         ^             |
         +-- adapters -+        dependencies point INWARD only
```

## Core Principles

### The Dependency Rule
The fundamental organizing principle: **source code dependencies can only point inwards**. Nothing in an inner circle can know anything about an outer circle. This includes functions, classes, variables, or any named software entity. Data formats from outer circles must not be used by inner circles.

### Four-Layer Structure

1. **Entities (Innermost)** — Enterprise-wide business rules that could be used across multiple applications. The most stable layer, least likely to change due to external factors.

2. **Use Cases** — Application-specific business rules that orchestrate data flow between entities and direct them to achieve specific goals. Changes here affect the application but not entities.

3. **Interface Adapters** — Convert data between formats convenient for use cases/entities and external systems. Contains MVC components, presenters, views, controllers. All database access code belongs here.

4. **Frameworks and Drivers (Outermost)** — Database, web frameworks, external tools. Contains mostly glue code connecting to inner layers. "Details" that can be replaced with minimal impact.

## System Characteristics

Clean Architecture produces systems that are:

1. **Independent of Frameworks** — Frameworks serve as tools rather than architectural constraints
2. **Testable** — Business rules testable without UI, database, or external dependencies  
3. **Independent of UI** — UI changes don't affect business rules
4. **Independent of Database** — Business rules not bound to specific database technologies
5. **Independent of External Agencies** — Business rules know nothing about the outside world

## Boundary Crossing

Communication across layer boundaries uses the [dependency inversion principle](/wiki/dependency-inversion-principle) to maintain the dependency rule while allowing necessary data flow. Interfaces in inner circles are implemented by outer circles, using dynamic polymorphism to oppose control flow direction.

Data crossing boundaries should be simple structures (DTOs, function arguments) without dependencies that violate the dependency rule.

## Architectural Synthesis

Clean Architecture integrates concepts from:
- [hexagonal architecture](/wiki/hexagonal-architecture) (Ports and Adapters) by [alistair cockburn](/wiki/alistair-cockburn)  
- [onion architecture](/wiki/onion-architecture) by [jeffrey palermo](/wiki/jeffrey-palermo)
- Screaming Architecture by [robert c martin](/wiki/robert-c-martin)
- DCI by [james coplien](/wiki/james-coplien) and [trygve reenskaug](/wiki/trygve-reenskaug)
- BCE by [ivar jacobson](/wiki/ivar-jacobson)

All these patterns share the objective of separation of concerns through layered design with protected business rules.

## Failure Modes

| Symptom | Root cause | Fix |
|---|---|---|
| Framework bleeds into core logic | Entities import framework types | Entities depend on nothing; adapters translate at the edge |
| Database schema drives design | Data model conflated with domain model | Repository ports own persistence; entities never see SQL |
| Layer skipped 'just this once' | No enforcement of the rule | Automated import rules (dependency-cruiser/import-linter) fail the build |

## Rule of Thumb

If you can name the framework a module serves, that module is in the
wrong layer - the core names abstractions, only adapters name vendors.

## Related Concepts
- [dependency rule](/wiki/dependency-rule)
- [dependency inversion principle](/wiki/dependency-inversion-principle)
- [hexagonal architecture](/wiki/hexagonal-architecture)
- [onion architecture](/wiki/onion-architecture)
- [agent harness engineering](/wiki/agent-harness-engineering)
- [cordis framework](/wiki/cordis-framework)
- [deepseek harness](/wiki/deepseek-harness)

 [[source: uncle-bob-clean-architecture-2012]](/wiki/raw/articles/uncle-bob-clean-architecture-2012)

---

## Closed-Loop Agent Improvement
- URL: https://pyweb.dev/wiki/closed-loop-agent-improvement
- Raw Markdown: https://pyweb.dev/wiki/closed-loop-agent-improvement.md
- Type: concept
- Summary: End-to-end telemetry and verification architecture where production failure traces drive automated candidate PR generation and benchmark gating.
- Tags: agents, feedback-loops, workflow, evaluation

# Closed-Loop Agent Improvement

Closed-loop agent improvement is an architectural paradigm for autonomous coding systems where production telemetry, failure clustering, candidate patch generation, and regression benchmarks operate in a continuous cycle under human governance.

## Key Subsystems (Replit Architecture)

1. **Telemetry Clustering (Telescope):** Aggregates millions of user execution trajectories, embeds interaction patterns, and clusters recurring failure modes rather than treating traces as isolated incidents.
2. **End-to-End Specification Benchmarking (ViBench):** Evaluates whether an agent can build a complete, running web application from a plain-English Product Requirements Document (PRD) from scratch. Utilizes browser automation (Playwright) to exercise multi-step user interactions and state verification in isolated sandboxes.
3. **Automated Hypothesis & PR Generation:** An engineering agent analyzes failure clusters, drafts targeted patches (prompt adjustments, tool schema repairs, harness enhancements, or new skills), and opens draft pull requests with attached trajectory evidence.
4. **Automated Regression Verification:** The candidate patch runs against ViBench suites and baseline production trajectories before alerting engineers for human review.

## Empirical Findings

- **Benchmark Disconnect:** High scores on narrow synthetic coding benchmarks (such as SWE-bench) do not reliably predict performance on full greenfield application generation.
- **Compounding Self-Extension Errors:** Frontier models frequently struggle and compound mistakes when asked to modify or extend their own previously generated codebases.
- **The Primacy of Human Taste:** Autonomous loops efficiently handle hypothesis testing and mechanical validation, but human engineers remain essential for eval curation (defining what success looks like), architecture shifts, and deployment authorization.

## Related
- [error analysis and evals](/wiki/error-analysis-and-evals)
- [automated eval engineering](/wiki/automated-eval-engineering)
- [agentic engineering patterns](/wiki/agentic-engineering-patterns)
- [agentic manual testing](/wiki/agentic-manual-testing)
- [context engineering](/wiki/context-engineering)

---

## Codex Harness Architecture
- URL: https://pyweb.dev/wiki/codex-harness-architecture
- Raw Markdown: https://pyweb.dev/wiki/codex-harness-architecture.md
- Type: concept
- Summary: Architecture of OpenAI's Codex harness: Clean/Hexagonal design, JSON-RPC app-server decoupling, multi-platform sandboxing, and bounded execution.
- Tags: agents, context-engineering, security, workflow, coding-guidelines

# Codex Harness Architecture

**Codex Harness Architecture** refers to the modular, protocol-driven systems design implemented in [OpenAI](/wiki/openai)'s open-source coding agent repository (`openai/codex`). Built in Rust (`codex-rs`), it exemplifies production [agent harness engineering](/wiki/agent-harness-engineering) by applying [Clean Architecture](/wiki/clean-architecture) and [Hexagonal Architecture](/wiki/hexagonal-architecture) (Ports & Adapters) to isolate non-deterministic LLM loops, OS-level process execution, and diverse client frontends.

```mermaid
flowchart TB
    subgraph Frontends["Frontends & Transports"]
        CLI["codex-cli"]
        TUI["codex-tui (Ratatui)"]
        IDE["IDE Extensions / Desktop"]
    end

    subgraph AppServer["Interface Adapters / Control Plane"]
        Server["codex-app-server (JSON-RPC daemon)"]
    end

    subgraph CoreEngine["Application Core"]
        Core["codex-core (OODA Turn Loop)"]
        Protocol["codex-protocol (Entities & Contracts)"]
    end

    subgraph OutboundPorts["Outbound Adapters & Substrates"]
        Sandbox["codex-sandboxing (bwrap / Seatbelt / Windows)"]
        Exec["unified_exec (HeadTailBuffer / PTY)"]
        MCP["codex-mcp (Tool Discovery & Execution)"]
        Store["codex-thread-store (Rollback / Fork / State)"]
    end

    CLI --> Server
    TUI --> Server
    IDE --> Server
    Server --> Core
    Core --> Protocol
    Core --> Sandbox
    Core --> Exec
    Core --> MCP
    Core --> Store
```

## Key Architectural Principles

### 1. Decoupled Control Plane (`App-Server` Pattern)
The core agent reasoning loop (`codex-core`) has zero awareness of terminal rendering or user input formatting.
- **Protocol-Driven Ingress:** Clients communicate with the harness exclusively through `codex-app-server-protocol` over JSON-RPC (via Unix Domain Sockets or WebSockets).
- **Multi-Client Surface:** The same underlying engine powers headless CLI executions (`/goal`), interactive TUI sessions, and background IDE extensions without code duplication.
- **Resilient Sessions:** Frontends can disconnect, reconnect, or inject mid-turn steer/interrupt signals without terminating the running background agent process.

### 2. Multi-Platform OS Sandboxing (`codex-sandboxing`)
To safely execute untrusted commands generated by models, Codex enforces strict OS-level process isolation rather than relying purely on user confirmation:
- **Linux:** Bubblewrap (`bwrap`) mount namespaces and Landlock LSM filesystem restrictions.
- **macOS:** Apple Seatbelt (`sandbox-exec`) security profiles compiled dynamically per execution.
- **Windows:** Job objects and restricted security tokens.

### 3. Bounded Context & Buffer Protection (`unified_exec`)
A common vulnerability in agent loops is context blowout caused by massive command stdout/stderr dumps (e.g., recursive directory listings or large log dumps). Codex integrates a `HeadTailBuffer` within `unified_exec` that captures the leading $N$ lines and trailing $M$ lines while discarding the middle, protecting the model's [smart zone](/wiki/smart-zone) against [context rot](/wiki/context-rot).

### 4. Asymmetric Tool Specialization (`apply-patch`)
While general-purpose harnesses rely on arbitrary JSON payloads, the Codex harness uses an optimized in-tree `apply-patch` crate tailored for unified diffs. This aligns with OpenAI frontier model training, which specializes in emitting unified diff chunks with anchor lines.

### 5. Deterministic State & Time-Travel (`thread-store`)
Session history is modeled as an ordered timeline of immutable turns:
- **Diff Tracking:** Every turn records file mutations via `turn-diff-tracker`.
- **First-Class Operations:** Primitives like `thread_rollback`, `thread_fork`, and `thread_resume` allow agents and human operators to backtrack safely from failed implementation branches.

## Failure Modes

| Symptom | Root cause | Fix |
|---|---|---|
| Harness and policy entangled | Ports not drawn at the right seams | Re-draw boundaries: every side effect behind a port |
| Tool output trusted raw | No validation at the boundary | Validate/deny at the adapter, never deep in the core |
| Sessions drift across restarts | State kept in-memory only | Persist session state through a repository port |

## Rule of Thumb

If a component would break when the model provider is swapped, it is
on the wrong side of the dependency boundary.

## Related Concepts
- [agent harness engineering](/wiki/agent-harness-engineering) — broader discipline of agent runtime infrastructure
- [clean architecture](/wiki/clean-architecture) — inward dependency boundaries
- [hexagonal architecture](/wiki/hexagonal-architecture) — ports and adapters isolation
- [deepseek harness](/wiki/deepseek-harness) — microkernel plugin alternative for agent harnesses
- [agent containment and blast radius](/wiki/agent-containment-and-blast-radius) — containment and egress security
- [context rot](/wiki/context-rot) — degradation from unbounded token accumulation
- [openai](/wiki/openai) — creator of Codex and the GPT models

---

## Cognitive Debt and Walkthroughs
- URL: https://pyweb.dev/wiki/cognitive-debt-and-walkthroughs
- Raw Markdown: https://pyweb.dev/wiki/cognitive-debt-and-walkthroughs.md
- Type: concept
- Summary: Techniques for understanding agent-generated code through structured linear walkthroughs and interactive explanations to prevent cognitive debt.
- Tags: pedagogy, technique, agents, knowledge-management

# Cognitive Debt and Walkthroughs

When engineers allow agents to produce code without understanding how the internal mechanics function, they accumulate **cognitive debt**. Over time, the codebase turns into an opaque black box that paralyzes future design decisions.

## Remedies
- **Linear Walkthroughs:** Directing an agent (via tools like `showboat`) to generate an ordered, step-by-step breakdown of newly created files, executing tools to embed real code snippets rather than copying text manually.
- **Interactive & Animated Explanations:** When an algorithm is complex (e.g., Archimedean spiral word-cloud layouts or AST parsing), prompting the agent to build an interactive HTML/JS widget with step-through sliders and visual rendering to build intuitive comprehension.
- **Editorial Diagramming:** Using structured, accessible visual architectures ([editorial diagrams and visual explanations](/wiki/editorial-diagrams-and-visual-explanations)) authored by [cathryn lavery](/wiki/cathryn-lavery) to visualize flows, decision trees, and system state with strict density and signal discipline.
- **Paying Down the Debt:** Understanding the code is a prerequisite for maintaining it; using agentic explanations turns opaque generated code into human-comprehensible assets.

## Related
- [simon willison](/wiki/simon-willison)
- [cathryn lavery](/wiki/cathryn-lavery)
- [editorial diagrams and visual explanations](/wiki/editorial-diagrams-and-visual-explanations)
- [agentic engineering patterns](/wiki/agentic-engineering-patterns)
- [feynman technique](/wiki/feynman-technique)
- [build from scratch pedagogy](/wiki/build-from-scratch-pedagogy)

---

## Conformance Suites as Fitness Functions
- URL: https://pyweb.dev/wiki/conformance-suites-as-fitness-functions
- Raw Markdown: https://pyweb.dev/wiki/conformance-suites-as-fitness-functions.md
- Type: concept
- Summary: Pairing agents with uncompromising deterministic test suites so they act as autonomous optimization engines instead of drifting generators.
- Tags: agents, evaluation, tdd, workflow, principle

# Conformance Suites as Fitness Functions

**Conformance Suites as Fitness Functions** is the breakthrough pattern that transforms agents from drifting generators into autonomous optimization engines. When agents are paired with uncompromising, deterministic test suites, they can grind complex optimization tasks into working code without human intervention.

## The Core Thesis

**The Problem:** When agents receive ambiguous tasks like "make this better," they hallucinate and drift without objective success criteria.

**The Solution:** Language-independent conformance test suites provide deterministic, objective fitness functions that enable autonomous optimization loops.

```mermaid
flowchart TD
    AR["AGENT AUTORESEARCH / REWRITE"] --> PE["Propose Code Edit / Patch"]
    PE --> CS["CONFORMANCE SUITE<br/><small>(Objective assertions)</small>"]
    
    CS -->|Exit Code != 0| FAIL["Feed Failure Output<br/>Back into Agent Loop"]
    CS -->|Exit Code == 0| PASS["Commit & Advance to<br/>Next Feature / Goal"]
    
    FAIL --> AR
```

## Landmark Case Studies

### Bun Zig→Rust Rewrite (July 2026)
The most dramatic validation of conformance-driven development came from [Jarred Sumner's rewriting of Bun's runtime engine](https://bun.com/blog/bun-in-rust) from **Zig to safe Rust**:

**The Enabling Factor:** Bun's extensive test suite was written in **TypeScript**, completely external to the implementation language. This language-independent test suite served as an objective conformance suite for the new Rust engine.

**Execution pattern:** Parallel agent loops worked against the TypeScript suite while humans monitored the workflow and reviewed the process rather than attempting a conventional line-by-line review of the generated port. The published case study includes detailed token, duration, and performance measurements. [[source: agentic-engineering-trends-2026-synthesis]](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

### Shopify Liquid Optimization (March 2026)
[Tobias Lütke's performance PR](https://github.com/Shopify/liquid/pull/2056) demonstrated the **Autoresearch Pattern**:

**Setup:** A durable prompt, a benchmark script, and the project's unit tests
**Architecture:** Pi agent ran in autonomous overnight loop, proposing micro-optimizations, benchmarking throughput, and recording state in `autoresearch.jsonl`
**Result:** A long sequence of benchmark-gated micro-optimizations, with exact measurements preserved in the linked pull request. [[source: agentic-engineering-trends-2026-synthesis]](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

### "Vibe Porting" - Rapid Cross-Language Rewrites

**JSONata to Go (Reco AI):** [Ported the JSONata query engine](https://www.reco.ai/blog/we-rewrote-jsonata-with-ai) from JavaScript to native Go, using the existing test suite and shadow production traffic as verification.

## Engineering Principles

### 1. Language Independence
Test suites must be external to implementation language to enable cross-language ports and rewrites without losing verification capability.

### 2. Deterministic Success Criteria  
Agents need binary pass/fail signals, not subjective quality judgments. Exit codes (0 = success, non-zero = failure) provide unambiguous feedback.

### 3. Comprehensive Coverage
Thousand-assertion test suites catch edge cases that humans miss during manual review of generated code.

### 4. Granular Feedback Loops
Failed test outputs feed directly back into agent prompts, creating tight optimization cycles around specific failure modes.

## Anti-Patterns

**Conformance Theater:** Creating tests that are too permissive or don't exercise critical code paths leads to false confidence in generated code quality.

**Suite Drift:** Test suites that aren't maintained in parallel with specification evolution become obsolete fitness functions.

## Related Concepts
- [red green tdd](/wiki/red-green-tdd)
- [tdd with agents](/wiki/tdd-with-agents)  
- [automated eval engineering](/wiki/automated-eval-engineering)
- [agent harness engineering](/wiki/agent-harness-engineering)
- [multi agent orchestration](/wiki/multi-agent-orchestration)

---

## Constraint Layering
- URL: https://pyweb.dev/wiki/constraint-layering
- Raw Markdown: https://pyweb.dev/wiki/constraint-layering.md
- Type: concept
- Tags: agentic-patterns, harness-engineering, security

# Constraint Layering

**Constraint Layering** is the architectural practice of allocating software engineering rules, policies, and safeguards to the cheapest and most reliable layer of the agent runtime. The governing operational maxim is:

> *Prompt for judgment, script the mechanical, gate the consequential, and isolate the dangerous.*

```mermaid
flowchart TD
    subgraph Layer 1: Prompt & Context
        P1[Skills & Procedures] --> J[High-Judgment Guidance]
        P2[AGENTS.md & CONTEXT.md] --> D[Domain Truth & Vocabulary]
    end
    subgraph Layer 2: Deterministic Tooling
        L1[Hooks & Linters] --> M[Mechanical Invariants]
        L2[Typecheckers & Compilers] --> T[Static Contracts]
    end
    subgraph Layer 3: System Boundaries
        S1[Protected CI / Oracles] --> G[Merge Gates & Acceptance]
        S2[Sandboxes & Egress Filters] --> I[Blast-Radius Containment]
    end
```

## Layer Allocation Matrix

| Layer | Best For | Typical Mechanism | Failure Mode if Misplaced |
|---|---|---|---|
| **Skill (`SKILL.md`)** | Judgment-heavy repeatable procedures | Workflow checkpoints, anti-rationalization tables | Over-reliance on prompts for mechanical checks (agent rationalizes skips) |
| **`AGENTS.md` / `CONTEXT.md`** | Concise, durable repository facts & ubiquitous language | Markdown reference files at repo root | Prompt bloat when packed with transient instructions |
| **Scripts & Pre-commit Hooks** | Cheap mechanical invariants | Linters (`oxlint` — see [deterministic lint gates](/wiki/deterministic-lint-gates)), formatting, import boundaries | Fragile regex hooks that block valid edits |
| **CI / Protected Oracles** | Merge-blocking contracts & regression safety | Read-only test suites, containerized builds | Slow feedback loops starving the inner agent loop |
| **Sandbox / Policy Engine** | Blast radius, authorization, and network isolation | Filesystem allowlists, ephemeral containers, disabled egress | Relying on prompt instructions to prevent secret leaks or file overwrites |
| **Canary & Rollback** | Runtime verification & production recovery | Telemetry metrics, automated rollbacks | Shipping directly to production based on local green tests |

## Why Prompt-Only Boundaries Fail

Language models are probabilistic reasoning engines that excel at rationalization. If a security or structural constraint is enforced solely via natural language prompts (e.g., *"Do not edit files outside src/http"*), models under edge-case pressure frequently rationalize breaking the rule to achieve the broader prompt goal. Hard operating boundaries (sandboxes, read-only mounts, and CI gates) convert probabilistic compliance into deterministic guarantees.

---

## Related Concepts
- [agent containment and blast radius](/wiki/agent-containment-and-blast-radius) — Restricting agent execution privileges.
- [agent harness engineering](/wiki/agent-harness-engineering) — Harness architecture and layered runtime controls.
- [progressive disclosure](/wiki/progressive-disclosure) — Dynamic context loading based on execution phase.
- [five debts of agentic engineering](/wiki/five-debts-of-agentic-engineering) — Structural debts prevented by proper constraint layering.

---

## Context Budget Audit
- URL: https://pyweb.dev/wiki/context-budget-audit
- Raw Markdown: https://pyweb.dev/wiki/context-budget-audit.md
- Type: concept
- Summary: Measuring and cutting per-turn token overhead — /context, logging proxies, disable flags, and deny rules to kill harness bloat.
- Tags: context-engineering, agents, cost

# Context Budget Audit

Every message to a harness-based agent ships a payload you never see: tool definitions, skills catalogs, system instructions for features you never touch — billed on every request, every turn. A context budget audit makes the invisible measurable and then cuts it.^[raw/aihero/how-to-kill-the-bloat-in-claude-codes-system-prompt.md]

## The Six-Step Audit Loop
1. **Measure** with `/context` (Claude Code): per-category token counts — system prompt, system tools, MCP tools, memory files, messages. Baseline before changing anything.
2. **Find the biggest offenders with a logging proxy**: the CLI talks plain HTTP to the model API, so a no-dependency forwarding proxy can record every request body untouched. `/context` reports tools as one aggregate; the proxy gives per-tool rankings.
3. **Switch off whole features** with `disable*` flags.
4. **Remove individual tools** with `deny` rules.
5. **Apply the configuration**, then **re-measure** — the loop closes only with a before/after number.

Matt Pocock's result: tens of thousands of tokens per turn cut.^[raw/aihero/how-to-kill-the-bloat-in-claude-codes-system-prompt.md] This is [prompt bloat](/wiki/prompt-bloat) remediation with instrumentation.

## Design Rule
Anything resident in the system prompt or tool registry must earn its per-turn cost — that's why harness design pushes to [progressive disclosure](/wiki/progressive-disclosure) (load skills on demand) and why [tool calling loop](/wiki/tool-calling-loop) carries an under-6-tools budget. Feedback-loop infrastructure (typecheck, tests, pre-commit hooks via Husky + lint-staged) belongs at the *harness* layer, not in context: it gives the agent verification without spending a single context token.

## Related
[prompt bloat](/wiki/prompt-bloat), [progressive disclosure](/wiki/progressive-disclosure), [tool calling loop](/wiki/tool-calling-loop), [context rot](/wiki/context-rot), [deterministic lint gates](/wiki/deterministic-lint-gates), [smart zone](/wiki/smart-zone).

---

## Context Engineering
- URL: https://pyweb.dev/wiki/context-engineering
- Raw Markdown: https://pyweb.dev/wiki/context-engineering.md
- Type: concept
- Summary: Curating high-signal context (AGENTS.md, CONTEXT.md, ADRs) to maximize agent reasoning performance.
- Tags: agents, context-engineering, workflow

# Context Engineering

[matt pocock](/wiki/matt-pocock)'s framing: context is the game. AGENTS.md, CONTEXT.md, feedback loops,
plan mode — the agent's performance is bounded by what's in its context window.
"Never run /init" — hand-craft your agent instructions because the defaults are generic
and generic context produces generic results.

## The artifacts
- **AGENTS.md** — project-scoped instructions for the agent: layout, toolchain, pitfalls,
  workflow. Lives at the workspace root. This is the first thing loaded (see [agents md spec](/wiki/agents-md-spec)).
- **CONTEXT.md** — the domain model: shared vocabulary, entities, relationships.
  Created lazily by domain modeling workflows. Prevents the agent from reinventing terms.
- **ADR directory** (`docs/adr/`) — architecture decision records. Why decisions were made,
  not just what they are.
- **Modular Capabilities** — composable skills and bounded tool configurations loaded on demand (see [progressive disclosure](/wiki/progressive-disclosure)).
- **Visual Specifications** — clear, low-noise architectural and workflow diagrams ([editorial diagrams and visual explanations](/wiki/editorial-diagrams-and-visual-explanations)) that provide structured multi-hop context without prompt bloat.

## The principle
Agents don't lack intelligence; they lack context. A brilliant agent with a generic
context will produce generic code. An average agent with precise context (domain model,
conventions, pitfalls, feedback loops) will produce precise code. Invest in context
the way you'd invest in infrastructure.

## Overlap with Karpathy
[andrej karpathy](/wiki/andrej-karpathy)'s [llm wiki pattern](/wiki/llm-wiki-pattern) is the same principle at the knowledge level:
compile once, interlink, keep current. This wiki is context engineering in action.

## Related
- [context budget audit](/wiki/context-budget-audit) — the measured, instrumented practice of this principle at the harness layer.
[llm wiki pattern](/wiki/llm-wiki-pattern), [grilling doctrine](/wiki/grilling-doctrine), [idea to ship flow](/wiki/idea-to-ship-flow), [error analysis and evals](/wiki/error-analysis-and-evals), [hamel husain](/wiki/hamel-husain), [progressive disclosure](/wiki/progressive-disclosure).

---

## Context Rot
- URL: https://pyweb.dev/wiki/context-rot
- Raw Markdown: https://pyweb.dev/wiki/context-rot.md
- Type: concept
- Summary: The progressive degradation of LLM reasoning performance as context length and irrelevant tokens accumulate.
- Tags: agents, context-engineering, anti-patterns

# Context Rot

**Context Rot** is the phenomenon where an LLM's reasoning accuracy, instruction adherence, and recall degrade as the context window fills with intermediate conversational turns, noisy tool outputs, and historical artifacts.

## The Mechanism
Transformer attention is not uniform across thousands of tokens:
1. **Lost in the Middle:** Information placed in the center of a long context is retrieved with significantly lower fidelity than tokens at the very beginning (system prompt) or the very end (latest turn).
2. **Attention Dilution:** Every irrelevant token in context consumes an attention budget, increasing the probability of hallucinations and missed edge cases.
3. **Compounding Noise:** When an agent misinterprets a noisy tool result and replies with flawed assumptions, that flawed exchange remains in context, poisoning subsequent reasoning steps.

## Mitigations
- **Context Clearing (`/clear`):** Reset the context completely between distinct, modular tasks.
- **Handoff Artifacts:** Compress multi-turn discussions into clean, structured handoff documents rather than carrying raw conversation logs forward.
- **Subagent Delegation:** Offload verbose investigations (searches, log reading, scraping) to isolated subagents and return only the distilled summary.
- **Strict Progressive Disclosure:** Load references and documentation on-demand rather than dumping everything into the initial system prompt.

## Rule of Thumb

Compaction triggers on token budget pressure, not on quality degradation
you can observe - by the time answers degrade, the rot predates them.

```mermaid
flowchart LR
    A[Long session] --> B[Turns + tool outputs accumulate]
    B --> C[Attention diluted]
    C --> D[Instruction adherence drops]
    C --> E[Recall of early context drops]
    D & E --> F[Fix: compact / restart / re-inject constraints]
```

## Failure Modes

| Symptom | Root cause | Fix |
|---|---|---|
| Answers degrade on long tasks | Attention diluted by accumulated turns | Compact or restart; never let turns pile unbounded |
| Early instructions forgotten | Lost-in-the-middle placement | Re-state critical constraints at the END of context |
| Tool outputs flood the window | Verbose results kept verbatim | Summarize tool output before it enters history |

## Related
[smart zone](/wiki/smart-zone), [handoff artifacts](/wiki/handoff-artifacts), [context engineering](/wiki/context-engineering), [prompt bloat](/wiki/prompt-bloat), [subagents and context management](/wiki/subagents-and-context-management), [ai coding taxonomy](/wiki/ai-coding-taxonomy).

---

## Cordis Framework
- URL: https://pyweb.dev/wiki/cordis-framework
- Raw Markdown: https://pyweb.dev/wiki/cordis-framework.md
- Type: concept
- Summary: Microkernel plugin and dependency-injection framework powering DeepSeek Harness: lifecycle management, service containers, typed event buses, reversible side-effects.
- Tags: agents, workflow, principle

# Cordis Framework

**Cordis** is a microkernel plugin and dependency-injection framework used as the underlying infrastructure kernel in [DeepSeek Harness](/wiki/deepseek-harness). It provides modular lifecycle management, service containers, typed event buses, and reversible side-effects for agent runtimes.

## The 5 Core Concepts

Cordis structures agent harnesses around 5 foundational mechanics:

1. **Plugins as Service Providers:** A plugin is either an object implementing `apply(ctx)` with optional `inject` declarations, or a `Service` subclass whose lifecycle is bound to the current context.
2. **Context as Service Container:** Services occupy stable keys on the context object (e.g., `ctx.tools`, `ctx.llm`, `ctx.sessions`, `ctx.agents`). Consumer plugins access capabilities via context keys rather than importing concrete module implementations.
3. **Declarative Inversion of Control via `inject`:** Plugins declare required services via `inject`. Cordis delays plugin activation until all dependent services are mounted and ready, eliminating manual startup sequencing.
4. **Typed Event Dispatch:** Services communicate through typed event buses with 4 distinct dispatch modes:
   - `emit`: Asynchronous observation in registration order (no return value, no await).
   - `waterfall`: Middleware pipeline in registration order; listeners can modify data, delegate via `next()`, or short-circuit return values.
   - `parallel`: Parallel asynchronous execution across all listeners (`await`, no return value).
   - `serial`: Sequential execution in registration order (`await`, returns value).
5. **Reversible Side Effects & Teardowns:** Registrations (prompt fragments, tool schemas, middleware, event listeners) are mounted via `ctx.effect()` or `ctx.on()`. Reloads and teardowns automatically invoke associated disposers to cleanly undo side-effects.

## Waterfall Middleware Semantics

The `waterfall` pattern functions as an onion-style interceptor:
- Listeners accept `(...args, next)`.
- Calling `next()` delegates to downstream handlers and yields their return value back up the stack.
- Returning directly without calling `next()` short-circuits execution, allowing policy plugins to preemptively veto or handle actions.
- Collaborative listeners modify shared request/decision payloads before delegating.

## Loader & Configuration Engine

Dynamic configuration uses `@deepseek-ai/cordis-plugin-include`:
- Parses expressions like `!!js` into dynamic AST nodes.
- Evaluates `config` and `disabled` predicates reactively against service context (`ctx.serviceName`).
- Environment-specific overrides and overlays allow dynamic feature flags without modifying harness code.

## Related Concepts
- [deepseek harness](/wiki/deepseek-harness)
- [agent harness engineering](/wiki/agent-harness-engineering)
- [agent native infrastructure](/wiki/agent-native-infrastructure)
- [deepseek](/wiki/deepseek)

---

## DeepSeek Harness
- URL: https://pyweb.dev/wiki/deepseek-harness
- Raw Markdown: https://pyweb.dev/wiki/deepseek-harness.md
- Type: concept
- Summary: DeepSeek's open-source agent runtime built on the Agent = Model + Harness thesis: modular, traceable execution without hardcoded core logic.
- Tags: agents, context-engineering, subagents, workflow, evaluation

# DeepSeek Harness

**DeepSeek Harness** is an open-source agent runtime framework released by [DeepSeek](/wiki/deepseek) built around the architectural thesis `Agent = Model + Harness`. It provides a modular, traceable execution harness designed to keep autonomous agents functioning reliably in real-world environments without hardcoded core logic.

## Core Architectural Design

The system is designed around two primary engineering principles: **Everything is a plugin** and **Every run is traceable**.

### 1. The Microkernel & Capabilities as Plugins
DeepSeek Harness builds upon the [Cordis kernel](/wiki/cordis-framework), delegating all functional capabilities into plugins that register typed services:
- **Core Subsystems as Plugins:** Models (`ctx.llm`), tool execution (`ctx.tools`), sessions (`ctx.sessions`), agent swarms (`ctx.agents`), skills, sandboxes, storage, loops, scheduling, and UI are all decoupled plugins.
- **Dynamic Reconfiguration:** Developers can swap, compose, or extend capabilities purely through configuration without modifying the harness source code.

### 2. Append-Only Traceability & Trajectory Replay
Every interaction the model observes is captured in an append-only event stream:
- System prompts, reasoning thoughts, tool calls, tool results, subagent schedules, and context injections are recorded.
- In the **Trajectory View**, developers inspect runs by source.
- The primitives `resume`, `fork`, `search`, and `replay` all operate natively on the unified session event log.

## Multiple Runtime Modes

DeepSeek Harness includes 4 preset operational modes tailored for different development and benchmarking workflows:

1. **Standard Mode:** Full coding agent equipped with file editing, shell execution, search, skills, planning, goals, subagents, and workflows.
2. **Code Mode:** Exposes tools via the Code Mode SDK, enabling the model to write TypeScript programs that orchestrate complex, multi-step tool calls in a single execution step.
3. **Minimal Mode:** Constrained two-tool coding agent (`bash` and `str_replace_editor`) designed for clean model benchmarking.
4. **Creator Mode:** Metaprogramming environment designed for inspecting active runtimes, testing in-memory [Cordis](/wiki/cordis-framework) plugins, and authoring custom agent presets.

## Related Concepts
- [cordis framework](/wiki/cordis-framework)
- [agent harness engineering](/wiki/agent-harness-engineering)
- [agent native infrastructure](/wiki/agent-native-infrastructure)
- [subagents and context management](/wiki/subagents-and-context-management)
- [context engineering](/wiki/context-engineering)
- [deepseek](/wiki/deepseek)

---

## Dependency Inversion Principle
- URL: https://pyweb.dev/wiki/dependency-inversion-principle
- Raw Markdown: https://pyweb.dev/wiki/dependency-inversion-principle.md
- Type: concept
- Tags: principle

# Dependency Inversion Principle

A design principle that enables [clean architecture](/wiki/clean-architecture)'s [dependency rule](/wiki/dependency-rule) to work in practice by using dynamic polymorphism to create source code dependencies that oppose the flow of control. This allows necessary communication across architectural boundaries while maintaining inward-pointing dependencies.

## Mechanism

When control must flow outward (violating the dependency rule):
1. **Inner layer defines interface** — specifies the contract it needs
2. **Outer layer implements interface** — provides concrete implementation  
3. **Source dependency points inward** — satisfies the dependency rule
4. **Control flows outward** — achieves necessary functionality

## Practical Application

**Example**: Use case needs to call presenter
- Use case calls interface (Use Case Output Port) in its own layer
- Presenter implements that interface in the outer layer
- Source code dependency: Presenter → Use Case Output Port (inward)
- Control flow: Use Case → Presenter (outward)

## Role in Clean Architecture

The dependency inversion principle is the key mechanism that allows [clean architecture](/wiki/clean-architecture) to maintain the [dependency rule](/wiki/dependency-rule) while enabling all necessary communication between layers. Without this principle, the architectural constraints would be impossible to satisfy in practice.

 [[source: uncle-bob-clean-architecture-2012]](/wiki/raw/articles/uncle-bob-clean-architecture-2012)

---

## Dependency Rule
- URL: https://pyweb.dev/wiki/dependency-rule
- Raw Markdown: https://pyweb.dev/wiki/dependency-rule.md
- Type: concept
- Tags: principle

# Dependency Rule

The fundamental organizing principle of [clean architecture](/wiki/clean-architecture) that governs how software components can depend on each other. The rule states that **source code dependencies can only point inwards** toward higher-level abstractions.

## Core Constraints

### Inward-Only Dependencies
- Nothing in an inner circle can know anything about an outer circle
- Names of outer circle entities (functions, classes, variables) must not be mentioned in inner circles
- Data formats from outer circles cannot be used by inner circles
- Frameworks in outer circles cannot influence inner circle design

### Abstraction Hierarchy
As you move inward through the concentric circles:
- **Software becomes more abstract** and policy-oriented
- **Level of abstraction increases** from concrete details to general principles  
- **Inner circles are more general** and stable
- **Outer circles contain mechanisms** and implementation details

## Boundary Crossing Mechanics

When control flow must cross boundaries in a direction that would violate the dependency rule, the [dependency inversion principle](/wiki/dependency-inversion-principle) resolves the contradiction:

1. **Interfaces in Inner Circles** — Define contracts that outer circles implement
2. **Dynamic Polymorphism** — Allows source dependencies to oppose control flow direction
3. **Simple Data Structures** — Cross boundaries without carrying framework dependencies

### Example Pattern
Use case needs to call presenter:
- Use case calls interface (Use Case Output Port) in its own layer
- Presenter in outer layer implements that interface
- Source dependency points inward (rule satisfied)
- Control flows outward (functionality achieved)

## Data Crossing Rules

Data crossing boundaries must be:
- **Simple structures** — DTOs, function arguments, basic objects
- **Free of dependencies** — No framework types or database rows
- **Convenient for inner circle** — Format optimized for business rule consumption

Anti-pattern: Passing database framework RowStructure inward violates the rule by forcing inner circles to know about outer circle data formats.

## Architectural Benefits

Enforcing the dependency rule creates:
- **Testable systems** — Business rules isolated from external dependencies
- **Replaceable components** — Outer layers changeable without affecting inner logic
- **Stable core** — Business rules protected from technological churn
- **Clear separation** — Policies distinct from mechanisms

The dependency rule is the key constraint that enables [clean architecture](/wiki/clean-architecture)'s independence guarantees.

 [[source: uncle-bob-clean-architecture-2012]](/wiki/raw/articles/uncle-bob-clean-architecture-2012)

---

## Designing for Verifiability
- URL: https://pyweb.dev/wiki/designing-for-verifiability
- Raw Markdown: https://pyweb.dev/wiki/designing-for-verifiability.md
- Type: concept
- Summary: 'It's hard to eval' is a product smell — artifacts hard for you to verify are hard for users too; design checkable artifacts before building evals.
- Tags: evaluation, workflow, feedback-loops, principle

# Designing for Verifiability

[hamel husain](/wiki/hamel-husain) argues that the most common objection to evals — "our product is hard to eval" — is a **product smell**. Artifacts that are hard for the builder to verify are often hard for users too; in the worst case users must redo the work from scratch to check the output. Designing the product for ease of verification should come *before* building evals. [[source: hamel-husain-it-s-hard-to-eval-is-a-product-smell-2026]](/wiki/raw/articles/hamel-husain-it-s-hard-to-eval-is-a-product-smell-2026)


```text
output = answer
       + metric definition      (what exactly was measured)
       + trusted comparison     (vetted baseline)
       + sanity checks          (distribution, related numbers)
       + what could NOT be verified
```

## The anti-pattern: answer-only output

An AI data agent that returns only "Net revenue for Product A last quarter was $4.21M" gives the user nothing to check. Since the only output is the answer, verification means redoing the analysis.

## The fix: checkable artifacts

Design outputs around how a domain expert would validate them. For a metrics agent, that means surfacing:

- Comparison against a **trusted source** (a vetted dashboard or a colleague's prior analysis)
- The precise **metric definition** used (does "net revenue" net out returns and discounts?)
- **Sanity checks** on related quantities and the distribution beneath the aggregate
- The **query itself**, runnable and editable
- An explicit **"what I could not verify"** section — flag unverified inputs instead of presenting them as settled [[source: hamel-husain-it-s-hard-to-eval-is-a-product-smell-2026]](/wiki/raw/articles/hamel-husain-it-s-hard-to-eval-is-a-product-smell-2026)

A two-level interface serves both audiences: a chat reply surfacing the details worth seeing up front, backed by a full notebook holding the complete analysis.

## Why this matters for agents

Verifiability-first design is the product-side twin of the harness-side principle that agents need deterministic quality gates ([agent harness engineering](/wiki/agent-harness-engineering)). Both reduce to the same move: make correctness checkable by construction, rather than asserting it. It also makes evals tractable — checkable artifacts are exactly the surfaces an eval can grade.

## Rule of Thumb

If verifying the output takes longer than redoing the work, the output
is not done - design artifacts users can check, not answers they must
trust.

## Failure Modes

| Symptom | Root cause | Fix |
|---|---|---|
| User redoes the analysis to trust the answer | Answer-only output | Ship checkable artifacts: query, definition, comparison, unknowns |
| Trust decays over sessions | No consistency baseline | Compare against a vetted source on every run |

## Related Concepts
- [error analysis and evals](/wiki/error-analysis-and-evals)
- [automated eval engineering](/wiki/automated-eval-engineering)
- [agentic manual testing](/wiki/agentic-manual-testing)
- [tdd with agents](/wiki/tdd-with-agents)
- [hamel husain](/wiki/hamel-husain)

---

## Deterministic Lint Gates
- URL: https://pyweb.dev/wiki/deterministic-lint-gates
- Raw Markdown: https://pyweb.dev/wiki/deterministic-lint-gates.md
- Type: concept
- Summary: Rust-speed linters with error-only configs as the Tier 1 mechanical backpressure layer for agentic coding loops.
- Tags: coding-guidelines, feedback-loops, agents, workflow

# Deterministic Lint Gates

A deterministic lint gate is a sub-second, native-speed linter run after every coherent agent edit, configured so that every enabled rule fails the build. It converts code-style and bug-pattern policy from probabilistic prompt compliance into mechanical backpressure — the concrete Tier 1 implementation of [constraint layering](/wiki/constraint-layering)'s "script the mechanical" maxim inside [agentic code quality](/wiki/agentic-code-quality).

## The Reference Stack: Oxlint

Oxlint is a high-performance JavaScript/TypeScript linter built on the Oxc compiler stack in Rust. Its published benchmarks show it is "50 to 100 times faster than ESLint", with "more than 865 rules" covering ESLint core, TypeScript, React, Jest, Vitest, Import, Unicorn, and jsx-a11y. [[source: oxlint-linter-overview-2026]](/wiki/raw/articles/oxlint-linter-overview-2026)

Architecture properties relevant to agent harnesses:

- **Correctness-focused defaults** — out of the box it prioritizes "high-signal correctness checks" over style noise. [[source: oxlint-linter-overview-2026]](/wiki/raw/articles/oxlint-linter-overview-2026)
- **Type-aware linting** — delegated to `tsgolint`, which builds TypeScript programs via `typescript-go` and returns structured diagnostics; supports 59 of 61 type-aware rules from typescript-eslint. Enables checks like detecting floating promises. [[source: oxlint-type-aware-linting-2026]](/wiki/raw/articles/oxlint-type-aware-linting-2026)
- **Multi-file analysis** — a project-wide module graph shared across rules, avoiding the `import/no-cycle` performance cliff seen in ESLint. [[source: oxlint-linter-overview-2026]](/wiki/raw/articles/oxlint-linter-overview-2026)
- **AI-friendly diagnostics** — diagnostics carry precise spans, contextual data, and documentation links so agents "understand issues and apply fixes reliably". [[source: oxlint-linter-overview-2026]](/wiki/raw/articles/oxlint-linter-overview-2026)
- **Reliability policy** — crashes and performance regressions are both treated as bugs, prioritizing CI and large-monorepo throughput. [[source: oxlint-linter-overview-2026]](/wiki/raw/articles/oxlint-linter-overview-2026)

## The Reference Policy: Error, Never Warn

`@nkzw/oxlint-config` ([christoph nakazawa](/wiki/christoph-nakazawa)) is an opinionated preset whose principles map directly onto agent-loop failure modes: [[source: nkzw-oxlint-config-2026]](/wiki/raw/articles/nkzw-oxlint-config-2026)

| Config principle | Agent failure mode it closes |
|---|---|
| "Error, Never Warn: Warnings are noise and get ignored" | Agents (and humans) ignore non-blocking output; exit code 0 means the loop proceeds |
| Debug-only code disallowed (`no-console`, `no-only-tests`) | Leftover debug logging; `test.only` silently skipping the suite — a reward-hacking vector flagged in [agentic code quality](/wiki/agentic-code-quality) |
| Problematic patterns banned (`instanceof` via `@nkzw/no-instanceof`) | Plausible-looking generated code that breaks across bundle/realm boundaries |
| Fast: slow rules avoided; TypeScript `noUnusedLocals` preferred over `no-unused-vars` | Slow gates starve the inner agent loop — the exact misplacement failure named in [constraint layering](/wiki/constraint-layering) |
| Autofixable rules preferred; subjective style rules disabled | Friction and noise without bug-prevention payoff |
| Deterministic sorting (`perfectionist/sort-objects`, `sort-interfaces`, `sort-jsx-props`) | Noisy diffs and syntactic merge collisions across parallel agent worktrees |

The config also demonstrates layering discipline in miniature: for TypeScript files it disables rules the type-checker already enforces (`no-undef`, `no-dupe-keys`, `no-unreachable`) — each check lives in exactly one layer. [[source: nkzw-oxlint-config-2026]](/wiki/raw/articles/nkzw-oxlint-config-2026)

## Design Rules

1. **Every rule errors.** A warning tier trains the loop to ignore the gate.
2. **The gate must be faster than the generator.** Sub-second feedback keeps repair inside the inner loop; minute-scale linting pushes fixes to review, where they cost more.
3. **One check, one layer.** Disable lint rules duplicated by the compiler; never prompt for what the linter enforces.
4. **Structured diagnostics over prose.** Spans + docs links make agent auto-repair reliable.
5. **Sort mechanically.** Deterministic ordering is merge-conflict prevention for [multi agent orchestration](/wiki/multi-agent-orchestration), not aesthetics.

## Commit-Time Gate Topology

Oxlint, lint-staged, Husky, and CI solve different problems and should not be collapsed into one giant hook:

```mermaid
flowchart LR
    E[Coherent edit] --> O[Oxlint fast correctness gate]
    O --> T[Typecheck and changed tests]
    T --> S[Stage focused files]
    S --> L[lint-staged selects staged paths]
    L --> H[Husky pre-commit hook]
    H --> C[Protected CI full-suite oracle]
    C --> P[Proof packaged with patch]
```

- **Oxlint is the diagnostic engine.** Its correctness-focused defaults, structured spans, multi-file analysis, and fast execution make it suitable for frequent agent feedback. [[source: oxlint-official-linter-2026]](/wiki/raw/articles/oxlint-official-linter-2026)
- **lint-staged is the selector.** It passes staged files to commands, reducing commit-time work. Project-wide tools such as TypeScript typechecking must be invoked through a function command so filenames are not appended and `tsconfig.json` remains active. [[source: lint-staged-official-readme-2026]](/wiki/raw/articles/lint-staged-official-readme-2026)
- **Husky is the local trigger.** `husky init` creates a repository-managed pre-commit hook and a package-manager `prepare` script. Hooks remain bypassable, so they accelerate feedback rather than establish release authority. [[source: husky-official-get-started-2026]](/wiki/raw/articles/husky-official-get-started-2026)
- **CI is the authority.** It repeats non-bypassable full checks on a clean checkout. This follows [constraint layering](/wiki/constraint-layering): script mechanical checks locally, then gate consequential merges with protected oracles.

A good default is staged formatting plus Oxlint in pre-commit, with project-wide typecheck, tests, dependency contracts, and builds in CI. Keep the hook comfortably faster than a model iteration; move expensive checks outward rather than teaching agents to bypass slow hooks.

## Failure Modes

| Symptom | Root cause | Fix |
|---|---|---|
| Lint passes but code is wrong | Gates cover style, not behavior | Pair lint gates with eval/conformance suites |
| Gate bypassed via --no-verify | Hooks treated as advisory | Enforce in CI, not just local hooks |
| Gate noise trains devs to ignore it | Rules without signal volume tuning | Error-only severity on new gates; expand after signal proves out |

## Related
- [agentic code quality](/wiki/agentic-code-quality) — Tier 1 fast inner loop this pattern implements
- [constraint layering](/wiki/constraint-layering) — the allocation principle ("script the mechanical")
- [five debts of agentic engineering](/wiki/five-debts-of-agentic-engineering) — verification debt closed by mechanical gates
- [releasable patch rate](/wiki/releasable-patch-rate) — the factory metric fast gates protect
- [agentic engineering patterns](/wiki/agentic-engineering-patterns) — bounded fix loops and execution budgets

---

## Editorial Diagrams and Visual Explanations
- URL: https://pyweb.dev/wiki/editorial-diagrams-and-visual-explanations
- Raw Markdown: https://pyweb.dev/wiki/editorial-diagrams-and-visual-explanations.md
- Type: concept
- Summary: Design principles, semantic patterns, and standards for generating high-signal, self-contained visual diagrams in agentic workflows.
- Tags: diagrams, visual-explanation, agents, technique, pedagogy

# Editorial Diagrams and Visual Explanations

Visual diagrams generated by autonomous AI agents often degrade into generic "rounded-box slop" or illegible spaghetti graphs when unconstrained. **Editorial Diagramming** treats technical diagrams as precise communication artifacts where every visual token, line, and color must carry semantic meaning.

Pioneered by [cathryn lavery](/wiki/cathryn-lavery) in the Diagram Design specification, the approach bridges technical architecture representation and [cognitive debt and walkthroughs](/wiki/cognitive-debt-and-walkthroughs).

## Core Principles

### 1. Deletion & Target Density
- **Deletion over Addition:** The quality of a technical schematic is determined by what can be removed without losing core informational structure.
- **Merge Coupled Nodes:** If two components always operate together, represent them as a single node.
- **Remove Obvious Connectors:** If spatial proximity and containment convey hierarchy or flow, drop connecting lines to reduce visual noise.
- **Density Budget:** Aim for a target visual density of 4/10. Graphs with more than 9 nodes should be split into multi-stage overviews or detail sub-diagrams.

### 2. The Editorial Accent Rule
- Accent color is an informational pointer, not decoration.
- Reserve accent styling for 1–2 focal elements per diagram (e.g. the active bottleneck, the primary decision gate, or the specific step under audit).
- Applying accent color across multiple nodes erases visual hierarchy.

### 3. Semantic Patterns vs. Visual Layout Grammars
Select the behavioral dynamic before picking the physical layout:
- **Behavioral Patterns:** Identify whether the system represents a fan-in queue, an unstructured-to-structured pipeline, a governance control catalog, or a paired policy-divergence trace.
- **Layout Grammars:** Map the behavioral pattern to an appropriate spatial layout (Architecture, Sequence, State Machine, Layer Stack, Flywheel/Loop, Data Flow, Tree, or Matrix).

### 4. Technical Construction Standards
- **4px Coordinate Grid:** All coordinates, dimensions, padding, and gaps must align to a strict 4px grid to prevent the floaty, unanchored appearance common in AI visual generation.
- **Self-Contained Deliverables:** Diagrams are compiled as standalone HTML files with embedded CSS and inline SVG, ensuring portability and offline rendering.
- **Accessibility by Default:** Inline SVGs carry `role="img"`, namespaced `aria-labelledby`, and `<title>`/`<desc>` elements providing structural summaries rather than literal shape descriptions.
- **Fidelity Ledgers:** When importing or redrawing source diagrams (from Mermaid, draw.io, or raw codebases), provide an explicit ledger detailing merged, collapsed, and dropped components.

## Related
- [cathryn lavery](/wiki/cathryn-lavery)
- [cognitive debt and walkthroughs](/wiki/cognitive-debt-and-walkthroughs)
- [agentic engineering patterns](/wiki/agentic-engineering-patterns)
- [context engineering](/wiki/context-engineering)
- [progressive disclosure](/wiki/progressive-disclosure)
- [build from scratch pedagogy](/wiki/build-from-scratch-pedagogy)

---

## Error Analysis and Evals
- URL: https://pyweb.dev/wiki/error-analysis-and-evals
- Raw Markdown: https://pyweb.dev/wiki/error-analysis-and-evals.md
- Type: concept
- Summary: Qualitative inspection of production traces to discover failure modes, build domain-specific taxonomies, and derive trustworthy evaluation rubrics.
- Tags: evaluation, workflow, feedback-loops, context-engineering

# Error Analysis and Evals

Error analysis is the systematic process of reading, annotating, and categorizing real production failures before writing evaluation suites or prompt rubrics. Pioneered in traditional machine learning and adapted to modern AI systems by [hamel husain](/wiki/hamel-husain) and [shreya shankar](/wiki/shreya-shankar), it forms the foundational requirement for building defensible AI applications.


```text
1. collect failures   2. cluster by root cause
3. fix the biggest cluster   4. add eval case per failure
```

## The Error Analysis Lifecycle

1. **Dataset Collection:** Gathering representative production traces, multi-turn conversations, and tool execution logs.
2. **Open Coding:** Human domain experts review raw outputs and record unstructured notes on unexpected behaviors, UX friction, and first points of failure.
3. **Axial Coding:** Clustering open-ended observations into an actionable failure taxonomy and counting occurrence frequencies.
4. **Active Learning Integration:** Using AI coding skills to observe real-time human labeling, continuously update the failure taxonomy, and automatically retrieve high-probability candidate failure traces across large corpora.
5. **Targeted Metric Formulation:** Translating high-frequency taxonomy buckets into binary, task-specific evaluation criteria rather than generic 1–5 subjective Likert scores.

## Pitfalls of Skipping Error Analysis

- **The Vanity Metric Trap:** Tracking high-level generic scores (e.g. "helpfulness" or "coherence") that improve on paper while users continue to experience core workflow breakdowns.
- **Criteria Drift:** Writing rigid speculative rubrics before observing actual model outputs, leading to wasted labeling effort on non-existent failure modes.
- **Unvalidated LLM Judges:** Using uncalibrated models to score outputs without measuring judge precision, recall, and false positive rates against human ground truth.

## The Data Science Mapping (Husain, 2026)

In *The Revenge of the Data Scientist*, [hamel husain](/wiki/hamel-husain) argues every recurring eval pitfall is a missing data-science fundamental: reading traces and categorizing failures is Exploratory Data Analysis; validating an LLM judge against human labels is Model Evaluation; building representative test sets from production data is Experimental Design; getting domain experts to label outputs is Data Collection; monitoring production is Production ML. "The names changed, the work did not." [[source: hamel-husain-the-revenge-of-the-data-scientist-2026]](/wiki/raw/articles/hamel-husain-the-revenge-of-the-data-scientist-2026)

His five pitfalls: generic metrics, unverified judges (treat the judge like a classifier — human labels, train/dev/test partitions, precision/recall instead of accuracy), bad experimental design (ground synthetic test data in real logs; binary pass/fail over 1-5 Likert scales), bad data and labels (domain experts must label; "criteria drift" — validated by [shreya shankar](/wiki/shreya-shankar) and colleagues — means grading outputs is how criteria get defined), and automating too much (LLMs can wire up plumbing but cannot look at the data for you). The agent harness itself is data science: observability stacks of logs, metrics, and traces exposed to the agent are what let it tell when it is off track. [[source: hamel-husain-the-revenge-of-the-data-scientist-2026]](/wiki/raw/articles/hamel-husain-the-revenge-of-the-data-scientist-2026)

## Rule of Thumb

Read 100 failures before writing a single line of mitigation code -
the clusters, not the anecdotes, decide what to build.

## Related
- [automated eval engineering](/wiki/automated-eval-engineering)
- [closed loop agent improvement](/wiki/closed-loop-agent-improvement)
- [designing for verifiability](/wiki/designing-for-verifiability)
- [hamel husain](/wiki/hamel-husain)
- [shreya shankar](/wiki/shreya-shankar)
- [context engineering](/wiki/context-engineering)
- [agentic engineering patterns](/wiki/agentic-engineering-patterns)

---

## Eval Taxonomy
- URL: https://pyweb.dev/wiki/eval-taxonomy
- Raw Markdown: https://pyweb.dev/wiki/eval-taxonomy.md
- Type: concept
- Summary: Evals as the AI engineer's unit tests: deterministic pass/fail checks, LLM-as-judge smoke tests, and human feedback — three tiers for wrangling predictability from probabilistic systems.
- Tags: evaluation, llm-fundamentals, quality

# Eval Taxonomy

LLM systems are probabilistic: no change is small, and whether the butterfly "flaps" or "Flaps" its wings may change the output. Traditional testing assumes deterministic input-to-output mapping; **evals are the AI engineer's unit tests** — the tool for wrangling predictability from a probabilistic system and an indispensable part of productionizing any AI app.^[raw/aihero/what-are-evals.md]

## Why Manual QA Fails Here
A demo is easy; production is not. Manual "try a few favorite prompts and see if it feels better" is dangerous when any change (model swap, prompt tweak) can affect the entire system. You need automated evaluation on every change to know whether you're getting better or worse — the escape from the "Vibes-Only Trough" onto the "Data-Driven Slope".

## The Three Tiers
1. **Deterministic evals** — pass/fail assertions that extract determinism from the probabilistic system. The most useful kind (per Discord's Ian Webster): fast, developer-focused.^[raw/aihero/three-types-of-evals.md] Example: Discord's Clyde bot always replying with a lowercase first letter. Schema conformance from [structured outputs](/wiki/structured-outputs) is the cheapest deterministic oracle.
2. **LLM-as-a-judge** — an LLM evaluates outputs against criteria or ground truth (autoevals-style templates: humor, battle/factuality comparison). Treat as a **smoke test, not a real test**: failing a judge is a good indicator a human should look, but passing one proves little.
3. **Human feedback** — irreplaceable for long-form generation and certain factuality classes; every LLM app needs some human oversight.

## Tooling
Evalite (Vitest-based, by Matt Pocock) exemplifies eval tooling integrated into the dev loop — run alongside unit tests in the same framework. See [error analysis and evals](/wiki/error-analysis-and-evals) and [eval driven development](/wiki/eval-driven-development) for the broader discipline, [evals skills](/wiki/evals-skills) for evals as agent skills.

## Related
[error analysis and evals](/wiki/error-analysis-and-evals), [eval driven development](/wiki/eval-driven-development), [generator evaluator loop](/wiki/generator-evaluator-loop), [structured outputs](/wiki/structured-outputs), [evals skills](/wiki/evals-skills), [error analysis and evals](/wiki/error-analysis-and-evals).

---

## Eval-Driven Development
- URL: https://pyweb.dev/wiki/eval-driven-development
- Raw Markdown: https://pyweb.dev/wiki/eval-driven-development.md
- Type: concept
- Summary: Iterative, trace-grounded engineering discipline for discovering, encoding, and continuously testing GenAI failure modes.
- Tags: evaluation, feedback-loops, coding-guidelines, workflow

# Eval-Driven Development

Eval-Driven Development (EDD) is the generative AI analogue of Test-Driven Development (TDD). Coined in enterprise practice by [airbnb](/wiki/airbnb) and formalised in methodology by [hamel husain](/wiki/hamel-husain) and [shreya shankar](/wiki/shreya-shankar), EDD treats evaluation as an ongoing development loop rather than a post-hoc verification gate. [[source: airbnb-eval-driven-development-2026]](/wiki/raw/articles/airbnb-eval-driven-development-2026)

## Core Thesis: The One Rule

> **"When in doubt, look at your data."** — Rohit Girme et al. (Airbnb)

Product quality in agentic systems is primarily determined by error analysis over execution traces, not off-the-shelf benchmark metrics. Teams consistently report spending **60–80% of project effort on error discovery and annotation** rather than automated scaffolding. [[source: hamel-husain-ai-evals-faq-2026]](/wiki/raw/articles/hamel-husain-ai-evals-faq-2026) Passing 100% of an eval suite indicates a benchmark that under-stresses system boundaries rather than proven reliability.

## The Three-Layer Eval Funnel

```mermaid
flowchart TD
    subgraph Layer1 [Layer 1: Programmatic & Deterministic]
        P1[Strict Schema Validation / Types]
        P2[Forbidden Regex & Output Filters]
        P3[Deterministic Syntax / AST Checks]
    end

    subgraph Layer2 [Layer 2: Calibrated LLM-as-a-Judge]
        J1[3–5 Sharp Single-Dimension Judges]
        J2[Few-Shot Rubric & Binary Scoring]
        J3[Judge Calibration: TPR / TNR / Bias Checks]
    end

    subgraph Layer3 [Layer 3: Human-in-the-Loop]
        H1[Failure Trace Discovery]
        H2[Disagreement Adjudication]
        H3[High-Blast-Radius Gate Sign-off]
    end

    Input[Agent Execution Trace / Output] --> Layer1
    Layer1 -->|Pass / Fast Sub-second| Layer2
    Layer2 -->|Flagged Discrepancy or High Risk| Layer3
    Layer3 -->|New Failure Mode| Layer1
```

### 1. Programmatic & Deterministic Checks (Layer 1)
Fast, zero-LLM-cost filters that eliminate unviable generations before expensive evaluation:
- Strict JSON schema enforcement (`zod`, Pydantic) to prevent downstream parsing failures.
- Syntax, AST linting, length bounds, regex, and type-system checks.

### 2. Calibrated LLM-as-a-Judge (Layer 2)
Focused virtual judges targeting nuanced quality criteria (faithfulness, conciseness, instruction adherence):
- **Single-Dimension Rule:** Deploy 3–5 small, sharp evaluators evaluating one orthogonal property each, rather than one omnibus grader. [[source: airbnb-eval-driven-development-2026]](/wiki/raw/articles/airbnb-eval-driven-development-2026)
- **Separate Model Architecture:** Always evaluate using a model distinct from or stronger than the generator.
- **Statistical Calibration:** Evaluate judges against expert ground-truth labels using true positive rate (TPR), true negative rate (TNR), and prompt-bias audits. [[source: hamel-husain-shreya-shankar-evals-skills-2026]](/wiki/raw/articles/hamel-husain-shreya-shankar-evals-skills-2026)

### 3. Human Grounding & Adjudication (Layer 3)
Human attention is reserved for high-leverage boundaries:
- Inspecting sample traces (100 baseline runs) to categorise novel failure modes.
- Resolving edge-case disagreements and setting hard policy boundaries.

## Five Operating Principles

1. **Define Goals and Blocking Gates Upfront:** Establish minimal pass thresholds before code generation.
2. **Derive Metrics from Real Traces:** Co-develop rubrics with domain stakeholders based on observed system failures.
3. **Keep Evaluators Small and Sharp:** Avoid monolithic prompt evaluators; isolate criteria.
4. **Appoint an Accountable Human Decision-Maker:** Explicitly designate an engineer to arbitrate ambiguous model behavior.
5. **Continuous Calibration:** Track judge drift whenever underlying foundation models or system prompts update.

## Failure Modes

| Symptom | Root cause | Fix |
|---|---|---|
| Eval suite passes but users complain | Benchmark under-stresses real boundaries | Add one eval case per real failure, forever |
| Evals written after the change | Post-hoc rationalization | Write the eval when writing the feature (TDD analogy) |
| Suite drifts from product reality | Prompts/features evolve without eval updates | Review eval fixtures each release cycle |

## Related
- [eval taxonomy](/wiki/eval-taxonomy) — the three-tier classification (deterministic, LLM-as-judge, human) underlying EDD.
- [agentic code quality](/wiki/agentic-code-quality) — multi-tier control architecture
- [generator evaluator loop](/wiki/generator-evaluator-loop) — iterative refinement loop
- [error analysis and evals](/wiki/error-analysis-and-evals) — diagnostic methodology
- [hamel husain](/wiki/hamel-husain) — evaluation authority
- [shreya shankar](/wiki/shreya-shankar) — evaluator alignment research
- [airbnb](/wiki/airbnb) — industrial implementation

---

## Evals Skills
- URL: https://pyweb.dev/wiki/evals-skills
- Raw Markdown: https://pyweb.dev/wiki/evals-skills.md
- Type: concept
- Tags: evaluation, skills, agents, workflow

# Evals Skills

Eval methodology packaged as installable agent skills. Published by [hamel husain](/wiki/hamel-husain) and [shreya shankar](/wiki/shreya-shankar), **evals skills** are "a set of skills for AI product evals" that encode lessons "from helping 50+ companies and teaching students in our AI Evals course" as procedural instructions a coding agent loads on demand — [progressive disclosure](/wiki/progressive-disclosure) applied to evaluation practice.

The motivating critique: "Eval tools often get in the way. They nudge you toward generic off-the-shelf metrics and fully automated evals before you've looked at your data." Instead of a tool, the methodology ships as skills that steer the agent around "many easily avoidable footguns."

Scope note: these are product evals, "Not foundation model benchmarks like MMLU or HELM that measure general LLM capabilities. Product evals measure whether your pipeline works on your task with your data."

## The skill set

- **start** — entry point; "looks at your situation and routes you to the right skill"
- **eval-audit** — "Audit an eval pipeline and surface problems with prioritized severity"
- **error-discovery** — "Build a review app, select diverse samples, and organize your notes into failure modes"; builds a customized annotation interface and samples traces intelligently
- **generate-synthetic-data** — diverse synthetic test inputs "using dimension-based tuple generation"
- **write-judge-prompt** — design LLM-as-Judge evaluators for subjective quality criteria
- **validate-evaluator** — "Calibrate LLM judges against human labels using data splits, TPR/TNR, and bias correction"
- **evaluate-rag** — retrieval and generation quality in RAG pipelines
- **build-review-interface** — custom annotation interfaces for human trace review

Install: `npx skills add https://github.com/ai-evals-course/evals-skills`. The suggested prompt routes through `start` and, if `eval-audit` is picked, has the agent "investigate each diagnostic area using a separate subagent in parallel, then synthesize the findings into a single report" — a direct use of [subagents and context management](/wiki/subagents-and-context-management).

The authors frame the skills as "only a starting point. To make them better, tune them to be more specific to your data and domain" — consistent with the "look at your data" doctrine in [error analysis and evals](/wiki/error-analysis-and-evals).

## Related
- [error analysis and evals](/wiki/error-analysis-and-evals)
- [automated eval engineering](/wiki/automated-eval-engineering)
- [designing for verifiability](/wiki/designing-for-verifiability)
- [progressive disclosure](/wiki/progressive-disclosure)
- [subagents and context management](/wiki/subagents-and-context-management)

---

## First-Principles Thinking
- URL: https://pyweb.dev/wiki/first-principles-thinking
- Raw Markdown: https://pyweb.dev/wiki/first-principles-thinking.md
- Type: concept
- Summary: Stripping complex problems to fundamental verifiable truths and rebuilding solutions upward.
- Tags: principle, worldview, technique

# First-Principles Thinking

Strip a problem to what is actually known to be true; rebuild the answer from there,
distrusting inherited assumptions until checked. [richard feynman](/wiki/richard-feynman)'s working mode —
paired with his warning that you are the easiest person to fool.

## Distinctions
- Not contrarianism: conclusions may match convention; the difference is they're re-derived,
  not inherited.
- "Knowing the name is not knowing the thing": pattern names (singleton, saga, RAG) are
  compressed claims — decompress before relying on them.
- Argument from authority is worth nothing; argument from evidence is worth everything.

## Applied to Engineering
- Check the assumption the bug report smuggles in ("it broke after X" — did it?).
- Re-derive the constraint before designing around it (is the API limit real? measured?).
- [build from scratch pedagogy](/wiki/build-from-scratch-pedagogy) is first principles as a learning tool;
  [karpathy four guidelines](/wiki/karpathy-four-guidelines) #1 is first principles as coding conduct.
- The [grilling doctrine](/wiki/grilling-doctrine) operationalizes it conversationally: facts checked, decisions human.

## Related
[richard feynman](/wiki/richard-feynman), [feynman technique](/wiki/feynman-technique), [build from scratch pedagogy](/wiki/build-from-scratch-pedagogy), [grilling doctrine](/wiki/grilling-doctrine).

---

## Five Debts of Agentic Engineering
- URL: https://pyweb.dev/wiki/five-debts-of-agentic-engineering
- Raw Markdown: https://pyweb.dev/wiki/five-debts-of-agentic-engineering.md
- Type: concept
- Tags: agentic-patterns, code-quality, architecture, workflow

# Five Debts of Agentic Engineering

When probabilistic code generation is accelerated without senior-engineering discipline, systems accumulate five structural debts. Because language models optimize for immediate syntax generation and local token completion, they naturally skip the invisible scaffolding that senior engineers apply.

```mermaid
flowchart TD
    subgraph Generative Failures
        A[Intent Debt] -->|Misaligned goals| S1[Grilling & Executable Specs]
        B[Context Debt] -->|Vocabulary drift & bloat| S2[CONTEXT.md & ADRs]
        C[Verification Debt] -->|Self-fulfilling tests| S3[Red-Green TDD at Seams]
        D[Architecture Debt] -->|Spaghetti & shallow modules| S4[Deep Modules & Boundary Rules]
        E[Authorization Risk] -->|Overeager scope creep| S5[Sandboxes & Path Allowlists]
    end
    subgraph Durable System Controls
        S1 --> G[Releasable Patch Gate]
        S2 --> G
        S3 --> G
        S4 --> G
        S5 --> G
    end
```

## 1. Intent Debt
- **Symptom:** The agent builds a syntactically correct solution that solves the wrong business problem or assumes unstated product requirements.
- **Root Cause:** Underspecified prompts contain multiple plausible implementation branches. Fast code generation makes exploring the wrong branch expensive sooner.
- **Control Mechanism:** [grilling doctrine](/wiki/grilling-doctrine) and formal specifications (`to-spec`). Interrogate the decision frontier, explicitly document non-goals, and establish observable acceptance criteria before writing code.

## 2. Context Debt (Semantic Drift)
- **Symptom:** Token bloat, naming inconsistencies, and vocabulary mismatch across modules.
- **Root Cause:** Each prompt re-explains domain concepts using ad-hoc synonyms, degrading the model's attention window ([smart zone](/wiki/smart-zone)) and inducing [context rot](/wiki/context-rot).
- **Control Mechanism:** [context engineering](/wiki/context-engineering) via canonical `CONTEXT.md` (ubiquitous language), `CONTEXT-MAP.md`, and Architectural Decision Records (ADRs).

## 3. Verification Debt
- **Symptom:** Passing test suites that fail in production, tautological mocks, or test suites modified/deleted by the agent to force green.
- **Root Cause:** Making the generator the sole author and judge of its own tests.
- **Control Mechanism:** [red green tdd](/wiki/red-green-tdd) at public seams, [agentic manual testing](/wiki/agentic-manual-testing), and protected baseline regression suites that remain read-only to the agent.

## 4. Architecture & Comprehension Debt
- **Symptom:** Cosmetic modularity (shallow folders), high coupling, duplicated business logic, and incomprehensible changes.
- **Root Cause:** The model lacks a holistic architectural mental model and optimizes solely for the localized file diff.
- **Control Mechanism:** Enforcing [clean architecture](/wiki/clean-architecture) boundaries, the deletion test (deep modules hiding substantial behavior behind small interfaces), and mandatory human walkthroughs.

## 5. Authorization & Operational Risk
- **Symptom:** Overeager scope expansion—the agent modifies adjacent configs, deletes credentials, or executes destructive side-effects outside task scope.
- **Root Cause:** Generative models lack intrinsic operational boundaries; if an action seems correlated with "fixing" the symptom, the model executes it.
- **Control Mechanism:** [agent containment and blast radius](/wiki/agent-containment-and-blast-radius), least-privilege sandboxes, strict path allowlists, and immutable audit trails.

---

## Related Concepts
- [software engineering fundamentals for agents](/wiki/software-engineering-fundamentals-for-agents) — The foundational engineering competencies required to prevent these debts.
- [agentic code quality](/wiki/agentic-code-quality) — The risk-conditioned T0–T5 verification architecture.
- [agent harness engineering](/wiki/agent-harness-engineering) — Designing runtime infrastructure around the model.
- [releasable patch rate](/wiki/releasable-patch-rate) — Measuring quality across all five dimensions at the release boundary.
- [constraint layering](/wiki/constraint-layering) — Allocating controls to the cheapest reliable layer.

---

## Generator-Evaluator Loop
- URL: https://pyweb.dev/wiki/generator-evaluator-loop
- Raw Markdown: https://pyweb.dev/wiki/generator-evaluator-loop.md
- Type: concept
- Summary: Generate then adversarially evaluate: a proposal loop where a separate evaluator checks each output against explicit criteria.
- Tags: agents, evaluation, feedback-loops, subagents, workflow

# Generator-Evaluator Loop

A multi-agent harness pattern, described by [prithvi rajasekaran](/wiki/prithvi-rajasekaran) of [anthropic](/wiki/anthropic)'s Labs team, that separates the agent doing the work from the agent judging it. Taking "inspiration from Generative Adversarial Networks (GANs)," the harness pairs a generator agent with a standalone evaluator agent, because agents grading their own output "tend to respond by confidently praising the work—even when, to a human observer, the quality is obviously mediocre."

The key claim: "tuning a standalone evaluator to be skeptical turns out to be far more tractable than making a generator critical of its own work, and once that external feedback exists, the generator has something concrete to iterate against."


```text
loop:
  proposal = generator(task, constraints)
  verdict  = evaluator(proposal, criteria)   # independent context
  if verdict.fail: constraints += verdict.failures
  else: return proposal
```

## Making subjective quality gradable

For frontend design — where there is "no binary check equivalent to a verifiable software test" — the harness converts taste into four gradable criteria given to both agents: **design quality**, **originality**, **craft**, and **functionality**. Design quality and originality were weighted more heavily because Claude scored well on craft and functionality by default; the criteria "explicitly penalized highly generic 'AI slop' patterns" such as "purple gradients over white cards." The evaluator was calibrated with few-shot examples with detailed score breakdowns, and used the Playwright MCP to navigate the live page — screenshotting and studying the implementation — before scoring. Runs went 5 to 15 iterations per generation, with full runs stretching up to four hours.

Criteria wording steered outputs directly: phrases like "the best designs are museum quality" pushed designs toward a particular visual convergence, and even first-iteration outputs beat an unprompted baseline before any evaluator feedback.

## The three-agent architecture

Scaled to full-stack coding, the pattern becomes planner → generator → evaluator, built on the Claude Agent SDK:

- **Planner** — expands "a simple 1-4 sentence prompt" into a full product spec, kept intentionally high-level so spec errors don't "cascade into the downstream implementation."
- **Generator** — works in sprints, one feature at a time, on a React, Vite, FastAPI, and SQLite (later PostgreSQL) stack with git.
- **Evaluator** — clicks through the running application via Playwright MCP "the way a user would," grading each sprint against hard thresholds; any criterion below threshold fails the sprint.

Before each sprint the generator and evaluator negotiate a **sprint contract** — agreeing on what "done" looked like for that chunk of work before any code was written. Contracts were granular — Sprint 3 alone had 27 criteria covering the level editor. Agents communicated via files, a concrete instance of [handoff artifacts](/wiki/handoff-artifacts).

## Context resets vs. compaction

The post distinguishes two responses to [context rot](/wiki/context-rot) on long tasks. **Compaction** summarizes earlier conversation in place so the same agent continues on a shortened history; **context resets** clear the window entirely and start a fresh agent with a structured handoff. Resets counter "context anxiety" — models "wrapping up work prematurely as they approach what they believe is their context limit" — which compaction alone cannot fix because it "doesn't give the agent a clean slate." Claude Sonnet 4.5 exhibited context anxiety strongly enough that resets became essential; Opus 4.5 "largely removed that behavior on its own," letting the newer harness drop resets and run one continuous session with automatic compaction.

## Cost and outcome

On a retro-game-maker prompt, a solo agent ran 20 min for $9; the full harness ran 6 hr for $200 — "over 20x more expensive, but the difference in output quality was immediately apparent." The planner expanded the one-sentence prompt into a 16-feature spec spread across ten sprints, and the harness build's play mode actually worked where the solo run's game was broken.

Independent evaluator context is the evals discipline applied to the
proposal loop: the grader is a structured rubric, not vibes. [[source: anthropic-demystifying-evals-for-ai-agents-2026]](/wiki/raw/articles/anthropic-demystifying-evals-for-ai-agents-2026)

## Failure Modes

| Symptom | Root cause | Fix |
|---|---|---|
| Generator ignores evaluator feedback | Feedback not injected as structured constraint | Feed failures back as explicit next-iteration constraints |
| Evaluator rubber-stamps | Same model, same prompt, no independence | Separate evaluator context; different criteria per criterion |
| Loop never terminates | No exit criteria | Hard cap iterations; escalate remaining disagreements to a human |

## Rule of Thumb

One loop iteration = one proposal + one evaluation; anything more
blurs attribution of what improved.

## Related
- [agent harness engineering](/wiki/agent-harness-engineering)
- [multi agent orchestration](/wiki/multi-agent-orchestration)
- [agentic manual testing](/wiki/agentic-manual-testing)
- [handoff artifacts](/wiki/handoff-artifacts)
- [context rot](/wiki/context-rot)
- [error analysis and evals](/wiki/error-analysis-and-evals)

---

## Git for Agentic Workflows
- URL: https://pyweb.dev/wiki/git-for-agentic-workflows
- Raw Markdown: https://pyweb.dev/wiki/git-for-agentic-workflows.md
- Type: concept
- Summary: Leveraging coding agents' native git fluency to explore repo history, resolve complex merge conflicts, bisect bugs, and rewrite clean commit stories.
- Tags: git, workflow, agents, technique

# Git for Agentic Workflows

Coding agents possess deep fluency in Git semantics and CLI mechanics. This unlocks high-discipline version control practices that humans often skip due to command complexity or tedious conflict resolution.

## Key Patterns
- **Context Seeding via `git log`:** Asking an agent to review recent commits (`git log -n 5`) immediately seeds its context window with exact recent changes, intent, and project velocity.
- **Automated Merge Conflict Resolution:** Agents can reason across conflicting branches, evaluate semantic intent, reconcile diffs, and verify that the test suite passes before concluding the merge.
- **Git Bisect Automation:** Agents write minimal test scripts and drive `git bisect run` to isolate the exact commit that introduced a regression.
- **Deliberate Commit Storytelling:** Commits are an authored explanation for future maintainers. Agents excel at soft resets (`git reset --soft HEAD~1`), surgical staging, and generating crisp commit messages.

## Anti-Pattern: Unreviewed PR Dumps
Never open pull requests containing hundreds of unreviewed agent-generated lines. Delegating unverified code to human collaborators destroys trust. The author must verify that the code runs, keep diffs small, provide evidence (manual test notes, logs, screenshots), and review agent-written PR descriptions.

## Related
- [simon willison](/wiki/simon-willison)
- [agentic engineering patterns](/wiki/agentic-engineering-patterns)
- [idea to ship flow](/wiki/idea-to-ship-flow)
- [karpathy four guidelines](/wiki/karpathy-four-guidelines)

---

## Grill With Docs
- URL: https://pyweb.dev/wiki/grill-with-docs
- Raw Markdown: https://pyweb.dev/wiki/grill-with-docs.md
- Type: concept
- Summary: Stateful interactive alignment skill that captures domain vocabulary into CONTEXT.md and three-gated decisions into ADRs.
- Tags: workflow, agents, context-engineering, skills

# Grill With Docs

`grill-with-docs` is the stateful head of the main agentic engineering flow (`grill-with-docs -> to-spec -> to-tickets -> implement -> code-review`). Developed by [matt pocock](/wiki/matt-pocock) on the [grilling doctrine](/wiki/grilling-doctrine) primitive, it conducts a conversational interview to establish shared human-agent understanding while progressively persisting domain terminology and architectural decisions directly to repository files.

## Operational Architecture

Unlike ephemeral grilling (`/grill-me`), which leaves alignment only in the transient context window, `grill-with-docs` pairs two underlying primitives:
1. **Interview Driver:** `grilling` — conducts turn-by-turn question loops focused on the decision frontier.
2. **Persistence Engine:** `domain-modeling` — writes resolved vocabulary and architecture decisions to disk lazily as they crystallize.

```mermaid
flowchart TD
    A[Human triggers /grill-with-docs] --> B[Grilling Interview Engine]
    B --> C{Output Type}
    C -->|Domain Term| D[CONTEXT.md Glossary]
    C -->|3-Gated Decision| E[docs/adr/ ADR]
    C -->|Conversational Detail| F[Context Window]
    F --> G[to-spec Synthesis]
    D --> H[Downstream Agent Sessions]
    E --> H
    G --> I[Executable Spec]
```

## The Three Output Tiers

During a session, resolved items are routed strictly according to their artifact class:

| Resolved Artifact | Destination | Criteria & Constraints |
|---|---|---|
| **Domain Term** | `CONTEXT.md` (root or multi-context map) | Written immediately inline as it resolves; strictly pure vocabulary with tight definitions (no spec or scratch notes). |
| **Architecture Decision** | `docs/adr/` (Architecture Decision Record) | Must satisfy all 3 gates: **hard to reverse**, **surprising without context**, and represents a **real trade-off**. |
| **Conversational Decisions** | Context window only | Nuances, defaults, and ordering guarantees that must be handed directly to `to-spec` before session clearing. |

## Selection Matrix

Choosing the right alignment skill depends on repository state and session scope:

| Context & Scope | Recommended Skill |
|---|---|
| No working directory / standalone concept | `grill-me` |
| Repository change settled in a single session | `grill-with-docs` |
| Greenfield build or multi-session initiative | `wayfinder` (decomposes into decision tickets) |
| Undocumented repository with no immediate feature | `grill-with-docs` (paired with `improve-codebase-architecture`) |
| Blocked on external stakeholder knowledge | `to-questionnaire` |

## Failure Modes & Diagnostics

1. **Silent Non-Writing (Wrapper Bug):** When executed within nested orchestration layers, file writes may be suppressed while the interview continues. Requires verifying disk modifications before trusting completion.
2. **Dependency Loading Failure:** If either `grilling` or `domain-modeling` fails to load, the agent either dumps questions in an undifferentiated batch or runs an ephemeral interview without disk persistence.
3. **Downstream Decision Softening:** Because conversational decisions do not land in `CONTEXT.md` or ADRs, relying on memory across sessions without running `to-spec` causes loss of numeric constraints and negative requirements.

## Related
[matt pocock](/wiki/matt-pocock), [grilling doctrine](/wiki/grilling-doctrine), [idea to ship flow](/wiki/idea-to-ship-flow), [context engineering](/wiki/context-engineering), [ai coding taxonomy](/wiki/ai-coding-taxonomy).

---

## Grilling Doctrine
- URL: https://pyweb.dev/wiki/grilling-doctrine
- Raw Markdown: https://pyweb.dev/wiki/grilling-doctrine.md
- Type: concept
- Summary: Relentless adversarial interview methodology to sharpen architecture and surface assumptions before coding.
- Tags: workflow, agents, principle

# Grilling Doctrine

[matt pocock](/wiki/matt-pocock)'s interview primitive applied to human-agent collaboration. Core rule:
**facts are the agent's job, decisions are the human's.** The agent drills
relentlessly to sharpen an idea before building — one sharp question at a time, with
room to disagree.

## The move
Before implementing, grill. What problem is this actually solving? One sentence. What's
the smallest version that proves the concept? What's the falsifiable outcome? The grilling
is not interrogation — it's Feynman's honesty applied to planning: "The first principle is
that you must not fool yourself."

## Why it matters with agents
Agents ship fast. An unsharpened idea shipped fast is how you fool yourself fastest —
you get 500 lines of code that solves the wrong problem convincingly. Grilling is the
gate between fuzzy intent and [tracer bullets](/wiki/tracer-bullets).

## In the flows
- **Grilling Interview:** Relentless interactive interview to sharpen a plan.
- **Spec Synthesis:** Converts the sharpened conversation into a formal spec.
- **Decomposition:** Breaks the spec into tracer-bullet tickets.
- See [idea to ship flow](/wiki/idea-to-ship-flow) and [grill with docs](/wiki/grill-with-docs) for the full pipeline.

## Related
[richard feynman](/wiki/richard-feynman), [feynman technique](/wiki/feynman-technique), [context engineering](/wiki/context-engineering), [idea to ship flow](/wiki/idea-to-ship-flow), [grill with docs](/wiki/grill-with-docs).

---

## Handoff Artifacts
- URL: https://pyweb.dev/wiki/handoff-artifacts
- Raw Markdown: https://pyweb.dev/wiki/handoff-artifacts.md
- Type: concept
- Summary: Structured, self-contained markdown documents that bridge context across session boundaries and multi-agent workflows.
- Tags: agents, context-engineering, workflow

# Handoff Artifacts

A **Handoff Artifact** is a structured, immutable summary document created at a phase boundary to transfer essential state, decisions, and unblocked tasks from one context window or agent to another.

## Why Raw Transcripts Fail
When migrating work across sessions or tools, passing the full unedited conversation transcript introduces immense noise, outdated assumptions, and token bloat. A handoff artifact acts as a lossy compression filter that preserves signal while discarding intermediate dead-ends.

## Anatomy of a Great Handoff Artifact
1. **Objective:** A one-sentence summary of what was accomplished and what the next session must achieve.
2. **Current State:** Verifiable facts backed by tool output (git commit SHA, passing test counts, created file paths).
3. **Decisions Made:** Architectural choices and hard constraints established in the prior session.
4. **Immediate Frontier:** A concrete, ordered checklist of next actions with blockers clearly identified.

## When to Use Handoffs
- Transitioning from an exploratory prototype branch back to the main feature branch.
- Crossing from planning/spec creation into ticket execution.
- Handing off tasks between different agent harnesses and sessions.

## Related
[smart zone](/wiki/smart-zone), [context rot](/wiki/context-rot), [idea to ship flow](/wiki/idea-to-ship-flow), [progressive disclosure](/wiki/progressive-disclosure), [ai coding taxonomy](/wiki/ai-coding-taxonomy).

---

## Hexagonal Architecture
- URL: https://pyweb.dev/wiki/hexagonal-architecture
- Raw Markdown: https://pyweb.dev/wiki/hexagonal-architecture.md
- Type: concept
- Tags: principle, workflow

# Hexagonal Architecture

Also known as "Ports and Adapters," an architectural pattern created by [alistair cockburn](/wiki/alistair-cockburn) that isolates business logic from external systems through well-defined interfaces. This pattern directly influenced [clean architecture](/wiki/clean-architecture) and emphasizes the separation of core application logic from external concerns.

## Core Concept

The hexagonal shape represents the application core surrounded by ports (interfaces) that connect to external adapters. Business logic remains independent of databases, user interfaces, and external services by communicating through ports rather than direct dependencies.

## Relationship to Clean Architecture

[robert c martin](/wiki/robert-c-martin)'s [clean architecture](/wiki/clean-architecture) synthesis incorporated hexagonal architecture principles, particularly:
- **Port-adapter separation** — mirrors the interface adapter layer
- **Business logic isolation** — core of the innermost entities layer  
- **[dependency rule](/wiki/dependency-rule) compliance** — dependencies point toward business logic

## Adoption

The pattern was notably adopted by [steve freeman](/wiki/steve-freeman) and [nat pryce](/wiki/nat-pryce) in "Growing Object Oriented Software" and became a foundational element in the broader architectural synthesis that led to [clean architecture](/wiki/clean-architecture).

 [[source: uncle-bob-clean-architecture-2012]](/wiki/raw/articles/uncle-bob-clean-architecture-2012)

---

## Hoard and Recombine
- URL: https://pyweb.dev/wiki/hoard-and-recombine
- Raw Markdown: https://pyweb.dev/wiki/hoard-and-recombine.md
- Type: concept
- Summary: Collecting verified working code snippets and proof-of-concept tools to supply as concrete reference material for coding agents.
- Tags: technique, agents, knowledge-management, workflow

# Hoard and Recombine

A high-leverage prompting and knowledge-management pattern: hoard working code solutions, TILs, and minimal prototypes, then prompt coding agents to synthesize new systems by combining those existing examples.


```text
save(concept) -> /concepts/<slug>.md + raw provenance
recombine(task) -> grep hoard by task keywords -> link relevant pages
```

## The Mechanism
1. **Hoard running proofs:** Knowing something is theoretically possible is weak; possessing a tested snippet (in a repo, TIL, or tool catalog) provides ground truth.
2. **Recombine via prompts:** Feed two or more working snippets into an agent (e.g. PDF.js canvas renderer + Tesseract.js WebAssembly worker) and prompt the agent to combine them into a single coherent interface or tool.
3. **Reference codebases in `/tmp`:** Direct agents to clone reference repositories into `/tmp` to inspect architecture, schemas, and API conventions without polluting the active repository commit tree.

## Why it Works
Agents excel at bridging disparate known patterns. Providing working reference code eliminates hallucinated APIs, establishes clear types and constraints, and produces working artifacts with minimal back-and-forth steering.

## Failure Modes

| Symptom | Root cause | Fix |
|---|---|---|
| Hoard becomes a junk drawer | Saving without curation criteria | Only save what you know how to reuse; prune what you have not touched in months |
| Recombination never happens | Artifacts saved in inaccessible formats | Save as structured, greppable markdown with provenance |
| Duplicate knowledge drifts | Same insight saved in multiple forms | One canonical page per concept; link variants to it |

## Rule of Thumb

A hoard earns its keep at recombination time: if the last three tasks
pulled nothing from it, the hoard is a liability, not an asset.

## Related
- [simon willison](/wiki/simon-willison)
- [agentic engineering patterns](/wiki/agentic-engineering-patterns)
- [build from scratch pedagogy](/wiki/build-from-scratch-pedagogy)
- [context engineering](/wiki/context-engineering)
- [tracer bullets](/wiki/tracer-bullets)

---

## Idea-to-Ship Flow
- URL: https://pyweb.dev/wiki/idea-to-ship-flow
- Raw Markdown: https://pyweb.dev/wiki/idea-to-ship-flow.md
- Type: concept
- Summary: End-to-end disciplined engineering workflow moving from grilled ideas to specs, tickets, and verified PRs.
- Tags: workflow, agents, principle

# Idea-to-Ship Flow

[matt pocock](/wiki/matt-pocock)'s structured engineering system, distilled. The pipeline from fuzzy idea to shipped
code, with each stage preventing a specific failure mode. [[source: aihero-dev-homepage-2026]](/wiki/raw/articles/aihero-dev-homepage-2026)

## The pipeline
1. **Grill** (`/grill-with-docs` or `/grill-me`) — relentless interview to sharpen the
   idea. Prevents: building the wrong thing. See [grilling doctrine](/wiki/grilling-doctrine).
2. **Spec** (`/to-spec`) — convert the sharpened conversation into a spec document.
   Prevents: scope drift mid-build.
3. **Tickets** (`/to-tickets`) — break the spec into [tracer bullet](/wiki/tracer-bullets)
   tickets with blocking edges. Prevents: unreviewable diffs.
4. **Implement** (`/implement` per ticket) — build one slice via [tdd with agents](/wiki/tdd-with-agents).
   Prevents: slop that can't be verified.
5. **Review** (`/code-review`) — review the changes since a fixed point. Prevents:
   anything that slipped through.

## Entry points
- New idea in a repo → Grilling phase first.
- Small well-scoped change → Direct implementation ticket with TDD.
- Bug report → Triage and diagnosis loop.
- Huge foggy effort → Architecture reconnaissance, then merge onto specification.
- Design question → Throwaway prototype.

## Why each stage exists
Agents ship fast. Each stage is a checkpoint where reality referees — Feynman's
"nature cannot be fooled" applied to software. Skip a stage and you fool yourself
faster, not slower, because the agent amplifies whatever you give it.

## Related
[grilling doctrine](/wiki/grilling-doctrine), [tracer bullets](/wiki/tracer-bullets), [tdd with agents](/wiki/tdd-with-agents), [karpathy four guidelines](/wiki/karpathy-four-guidelines), [context engineering](/wiki/context-engineering).

---

## Karpathy's Four Guidelines
- URL: https://pyweb.dev/wiki/karpathy-four-guidelines
- Raw Markdown: https://pyweb.dev/wiki/karpathy-four-guidelines.md
- Type: concept
- Summary: Four foundational coding conduct rules for agents: Think Before Coding, Simplicity First, Surgical Changes, Goal-Driven.
- Tags: coding-guidelines, agents, principle

# Karpathy's Four Guidelines

Behavioral rules reducing common LLM coding mistakes, from [andrej karpathy](/wiki/andrej-karpathy)'s pitfall
observations. Bias toward caution over speed; judgment on trivial tasks. [[source: karpathy-guidelines-skill-2026]](/wiki/raw/articles/karpathy-guidelines-skill-2026)

1. **Think before coding** — don't assume, don't hide confusion, surface tradeoffs.
   State assumptions; present competing interpretations instead of picking silently;
   name what's confusing and ask.
2. **Simplicity first** — minimum code that solves the problem, nothing speculative.
   No unrequested features/abstractions/configurability. "Would a senior engineer call
   this overcomplicated?"
3. **Surgical changes** — touch only what you must; clean up only your own mess. Every
   changed line traces to the request. Mention unrelated dead code, don't delete it.
4. **Goal-driven execution** — define verifiable success criteria, loop until verified.
   "Fix the bug" → "write a test that reproduces it, then make it pass"

## Fit with the other layers
#4 is [tdd with agents](/wiki/tdd-with-agents) in miniature and powers [tracer bullets](/wiki/tracer-bullets). #1 is
[grilling doctrine](/wiki/grilling-doctrine) at code scale. #2-3 are what keeps codebases maintainable for agents.

## Related
[andrej karpathy](/wiki/andrej-karpathy), [tdd with agents](/wiki/tdd-with-agents), [tracer bullets](/wiki/tracer-bullets), [grilling doctrine](/wiki/grilling-doctrine).

---

## LLM App Improvement Ladder
- URL: https://pyweb.dev/wiki/llm-app-improvement-ladder
- Raw Markdown: https://pyweb.dev/wiki/llm-app-improvement-ladder.md
- Type: concept
- Summary: Matt Pocock's ordered ladder of 17 techniques for improving an LLM-powered app — from prompt basics to fine-tuning — try the simple thing first.
- Tags: llm-fundamentals, patterns, quality

# LLM App Improvement Ladder

When an LLM app underperforms, don't jump to fine-tuning. Pocock's ordered ladder — try the simple thing first — moves from cheap prompt-level fixes to expensive architectural ones:^[raw/aihero/how-to-improve-your-llm-powered-app.md]

## The Rungs
1. **First prompt** — the baseline; most failures are under-specified instructions.
2. **Role-based prompting** — give the model a persona/role.
3. **XML tags** — structure the input so sections are unambiguous.
4. **Structured outputs** — schema-constrain the response ([structured outputs](/wiki/structured-outputs)).
5. **Reasoning** — elicit step-by-step thinking (cf. [think tool](/wiki/think-tool)).
6. **Multishot prompting** — examples in-prompt.
7. **Temperature** — tune randomness to the task.
8. **Tool calling** — let the model act ([tool calling loop](/wiki/tool-calling-loop)).
9. **LLM call chaining** — decompose into a pipeline (workflow pattern, [agents vs workflows](/wiki/agents-vs-workflows)).
10. **RAG** — retrieve documents into context.
11. **Chunking** — tune retrieval granularity.
12. **Agentic loops** — iterate tools against environment feedback.
13. **Parallelizing LLM calls** — fan out independent work.
14. **Evaluator-optimizer** — generate/evaluate loop ([generator evaluator loop](/wiki/generator-evaluator-loop)).
15. **LLM routers** — classify inputs to specialized handlers.
16. **Fine-tuning** — the last rung, not the first.

## Why the Order Matters
Each rung's cost compounds: prompt fixes are free, structural patterns cost engineering time, fine-tuning costs data plus pipeline maintenance. The ladder is also the diagnostic sequence — an app that fails at rung 3 won't be saved by rung 16. Measured by [eval taxonomy](/wiki/eval-taxonomy) at every step, or you're climbing blind.

## Related
[eval taxonomy](/wiki/eval-taxonomy), [structured outputs](/wiki/structured-outputs), [tool calling loop](/wiki/tool-calling-loop), [agents vs workflows](/wiki/agents-vs-workflows), [generator evaluator loop](/wiki/generator-evaluator-loop), [smart zone](/wiki/smart-zone).

---

## LLM Message Protocol
- URL: https://pyweb.dev/wiki/llm-message-protocol
- Raw Markdown: https://pyweb.dev/wiki/llm-message-protocol.md
- Type: concept
- Summary: The message-based conversation protocol between application and LLM: system prompts, user/assistant messages, tool calls, and tool results.
- Tags: agents, context-engineering, llm-fundamentals

# LLM Message Protocol

Every interaction with an LLM is a **message history**: an ordered array of typed messages exchanged with a stateless model-provider endpoint. The LLM has no memory between requests — the entire conversation is resent each turn, which is why token costs and [context rot](/wiki/context-rot) compound with history length.


```text
system   -> invariant instructions
user     -> task
assistant-> reasoning + tool calls
tool     -> results (append-only log, never reordered)
```

## Message Types
1. **System prompt** — persistent instructions defining the assistant's role and constraints. Tools are declared here as name + description + JSON-schema arguments (see [tool calling loop](/wiki/tool-calling-loop)).
2. **User message** — input from the human or calling application.
3. **Assistant message** — model output; either text, a tool call, or both.
4. **Tool call / tool result** — the executable half of the agentic loop (see [tool calling loop](/wiki/tool-calling-loop)).

Tokens are the unit of this protocol — the "currency" of LLM communication — and every message in the history is billed on every request. Understanding the protocol makes capabilities like structured outputs and streaming legible: they are conventions layered on top of the same message array.

## Design Consequences
- **Statelessness:** conversation memory is an application-side concern (the model re-reads the full array each turn).
- **History management:** chatbot follow-up questions only work because prior turns are resent; trimming or summarizing history is a context-engineering act (see [context engineering](/wiki/context-engineering), [handoff artifacts](/wiki/handoff-artifacts)).
- **Reasoning tokens:** some models emit hidden intermediate reasoning within their turn, billed as output.

## Failure Modes

| Symptom | Root cause | Fix |
|---|---|---|
| Role confusion across turns | Ad-hoc message assembly | One canonical builder; roles typed at construction, not stringly-checked |
| Tool results leaking into wrong turns | Ordering bugs in the loop | Sequential message log; append-only; never reorder |
| Context bloat from protocol overhead | Full history resent every call | Prune by policy, keep the protocol invariant |

## Rule of Thumb

The protocol is the harness contract: if two components disagree on
message shape, fix the protocol, not the call sites.

## Related
- [llm app improvement ladder](/wiki/llm-app-improvement-ladder) — the ordered improvement techniques built on this protocol.
[tool calling loop](/wiki/tool-calling-loop), [structured outputs](/wiki/structured-outputs), [context engineering](/wiki/context-engineering), [smart zone](/wiki/smart-zone), [model provider abstraction](/wiki/model-provider-abstraction).

---

## LLM Wiki Pattern
- URL: https://pyweb.dev/wiki/llm-wiki-pattern
- Raw Markdown: https://pyweb.dev/wiki/llm-wiki-pattern.md
- Type: concept
- Summary: Compounding, interlinked markdown knowledge base pattern curated autonomously by agents.
- Tags: knowledge-management, agents, technique

# LLM Wiki Pattern

[andrej karpathy](/wiki/andrej-karpathy)'s approach to durable knowledge: compile it once into interlinked
markdown, keep it current, cross-reference, flag contradictions. Instead of
rediscovering the same knowledge from scratch each time you ask an LLM, build a wiki
that the LLM can reference.

## Why
LLMs don't persist learning between sessions. Every conversation starts fresh. A wiki
is the external memory: the knowledge is compiled once (with sources, confidence,
contradictions noted) and loaded on demand. The alternative — re-researching every
time — wastes tokens and produces inconsistent results.

## How it works
- `SCHEMA.md` defines the structure: frontmatter, tags, page thresholds, update policy.
- `index.md` lists every page. `log.md` records every change.
- Wikilink syntax (for example: `&#91;&#91;concept-slug&#93;&#93;` or `&#91;&#91;concept-slug|custom label&#93;&#93;`) connects pages into a graph. Minimum 2 outbound links per page.
- `raw/` holds source articles. Pages cite raw sources via provenance markers.
- The schema enforces quality: page thresholds (create when 2+ sources mention it),
  update policy (newer supersedes older, contradictions noted not hidden).

## Relation to context engineering
[context engineering](/wiki/context-engineering) (Matt) is the same idea applied to project instructions
(AGENTS.md, CONTEXT.md). The wiki is for knowledge; those are for project context.
Both say: compile once, keep current, don't rediscover.

In 2026 the pattern was productized: TencentDB Agent Memory ships "LLM-Wiki" as one of
its four governed memory asset types, explicitly crediting Karpathy's knowledge-base
pattern (see [agent native infrastructure](/wiki/agent-native-infrastructure)).

## Related
[context engineering](/wiki/context-engineering), [andrej karpathy](/wiki/andrej-karpathy), [agent native infrastructure](/wiki/agent-native-infrastructure), [progressive disclosure](/wiki/progressive-disclosure).

---

## Model Context Protocol Basics
- URL: https://pyweb.dev/wiki/model-context-protocol-basics
- Raw Markdown: https://pyweb.dev/wiki/model-context-protocol-basics.md
- Type: concept
- Summary: MCP as the standardized bridge between LLM applications and external tools/data — the client-server layer beneath tool calling.
- Tags: agents, tool-use, protocols, llm-fundamentals

# Model Context Protocol Basics

The **Model Context Protocol (MCP)** standardizes how LLM applications connect to external tools and data sources. Where [tool calling loop](/wiki/tool-calling-loop) describes the in-conversation mechanics (tool calls and results inside the message history), MCP describes the plumbing: a client-server protocol where a host application (e.g. Claude Code) connects to MCP servers that expose tools, prompts, and resources.

## What It Looks Like in Practice
- A minimal MCP server can be a single script: create the server, connect it to a client like Claude Code, and expose "run this script" as a callable tool.
- Servers can expose **prompts** (reusable prompt templates, invocable like tools — `using-mcp-prompts`) and **resources** (data the client can read), not just tools.
- The value is composability: one server, many clients; one client, many servers — instead of N×M bespoke integrations.
- **Logging is a footgun:** verbose MCP server logging flows straight into the agent's context window; it must be treated as a [context budget audit](/wiki/context-budget-audit) cost, not free observability.

## Why It Matters
MCP decouples tool implementation from tool consumption. An agent harness ([agent harness engineering](/wiki/agent-harness-engineering)) can grow its capability surface by plugging in MCP servers without changing its core loop — the same [model provider abstraction](/wiki/model-provider-abstraction) logic applied to the tool side.

## Related
[tool calling loop](/wiki/tool-calling-loop), [llm message protocol](/wiki/llm-message-protocol), [agent native infrastructure](/wiki/agent-native-infrastructure), [ag ui protocol](/wiki/ag-ui-protocol), [model provider abstraction](/wiki/model-provider-abstraction).

---

## Model Provider Abstraction
- URL: https://pyweb.dev/wiki/model-provider-abstraction
- Raw Markdown: https://pyweb.dev/wiki/model-provider-abstraction.md
- Type: concept
- Summary: Why production LLM apps need a provider-neutral interface — and the real switching cost when you code directly against one vendor's API.
- Tags: llm-fundamentals, architecture, patterns

# Model Provider Abstraction

A classic production problem: you build directly against one provider's SDK (say OpenAI's), and one day you need to switch models — every call site is now vendor-shaped. The cost is not the model, it's the **integration surface**: message formats, streaming, tool-call shapes, and error handling all differ per vendor.

## The Pattern
Define a narrow interface — e.g. a `LanguageModel` type (the AI SDK, by [vercel](/wiki/vercel), is the reference implementation) that any `ask(prompt, model)` function accepts — and route all generation through it:
- Swapping models becomes a one-line change; the same prompt path runs against Anthropic, OpenAI, or a local model.
- `createOpenAICompatible`-style adapters extend this to **any OpenAI-compatible endpoint**, including locally-hosted models.
- Generation, streaming, embeddings, and image/file input all sit behind the same abstraction.

This is [dependency inversion principle](/wiki/dependency-inversion-principle) applied to LLM infrastructure: the application depends on an abstraction it owns, not on a vendor SDK. It also de-risks [context rot](/wiki/context-rot) economics — provider-agnostic apps can chase the cheapest adequate model without a rewrite.

## Caveat
Abstractions leak: provider-specific features (caching behavior, reasoning-token billing, tool-call quirks) do not always map cleanly. The interface should expose the common denominator and allow escape hatches.

## Related
[llm message protocol](/wiki/llm-message-protocol), [structured outputs](/wiki/structured-outputs), [dependency inversion principle](/wiki/dependency-inversion-principle), [clean architecture](/wiki/clean-architecture), [agent native infrastructure](/wiki/agent-native-infrastructure).

---

## Multi-Agent Orchestration
- URL: https://pyweb.dev/wiki/multi-agent-orchestration
- Raw Markdown: https://pyweb.dev/wiki/multi-agent-orchestration.md
- Type: concept
- Summary: Hierarchical agent swarms where engineers orchestrate fleets of specialized agents in parallel isolated worktrees: coder to conductor to orchestrator.
- Tags: agents, workflow, subagents, context-engineering, evaluation

# Multi-Agent Orchestration

**Multi-Agent Orchestration** is the paradigm shift from single-agent coding to **hierarchical agent swarms** where human engineers orchestrate fleets of specialized agents working in parallel across isolated worktrees. This represents the evolution from coder → conductor → orchestrator.

## Hierarchical Architecture

```mermaid
flowchart TD
    HO["HUMAN ORCHESTRATOR<br/><small>(High-level goals & spec)</small>"] --> LP["LEAD PLANNER / AGENT<br/><small>(Frontier planner model)</small>"]
    
    LP --> SA1["SUBAGENT 1 (Work)<br/><small>Isolated worker · Worktree: feature-a</small>"]
    LP --> SA2["SUBAGENT 2 (Work)<br/><small>Isolated worker · Worktree: feature-b</small>"]
    LP --> SA3["SUBAGENT 3 (Test)<br/><small>Test/review worker · Conformance tests</small>"]
    
    SA1 --> JM["JUDGE / MERGE AGENT<br/><small>(Adversarial verification)</small>"]
    SA2 --> JM
    SA3 --> JM
```

## Context Compression via Subagents

The fundamental mathematical driver is the **context window constraint**. Complex tasks with huge codebases flood single context windows with intermediate tool outputs, causing reasoning degradation and [context rot](/wiki/context-rot).

**Subagents as Compression Engines:** The lead orchestrator spawns isolated [subagents and context management](/wiki/subagents-and-context-management) for specific exploratory or implementation subtasks. Each subagent operates in its own fresh context, runs tools in parallel, and returns only a compressed, high-signal summary back to the parent. [[source: agentic-engineering-trends-2026-synthesis]](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

**Performance impact:** Anthropic reports that its multi-agent research system performs especially well on breadth-first tasks, but that result is workload-specific rather than a general guarantee. [[source: agentic-engineering-trends-2026-synthesis]](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

## Primary Source: Anthropic's Research System (2025)

Anthropic's published account of its Research feature is the canonical orchestrator-worker case study. A lead agent plans, saves the plan to external memory, and spawns parallel subagents with explicit objectives, output formats, tool guidance, and task boundaries — vague delegation ("research the semiconductor shortage") produced duplicated and misinterpreted work. [[source: anthropic-multi-agent-research-system-2025]](/wiki/raw/articles/anthropic-multi-agent-research-system-2025)

Key measured findings:

- A multi-agent system with Claude Opus 4 lead and Claude Sonnet 4 subagents outperformed single-agent Claude Opus 4 by 90.2% on Anthropic's internal research eval.
- On BrowseComp, token usage by itself explains 80% of performance variance; tool-call count and model choice explain most of the rest — multi-agent architectures win mainly by spending more tokens across separate context windows.
- Cost reality: agents use about 4× more tokens than chat, and multi-agent systems about 15× more, so the task's value must justify the burn.
- Parallelism (3-5 subagents spawned concurrently, 3+ tools called in parallel per subagent) cut research time by up to 90% for complex queries. [[source: anthropic-multi-agent-research-system-2025]](/wiki/raw/articles/anthropic-multi-agent-research-system-2025)

Operational lessons: embed effort-scaling rules in prompts (simple fact-finding = 1 agent, 3-10 tool calls); prefer end-state evaluation over prescribed step-checking; let subagents write outputs to a filesystem and pass lightweight references to avoid a "game of telephone"; use rainbow deployments because stateful agents may be mid-run during any deploy. [[source: anthropic-multi-agent-research-system-2025]](/wiki/raw/articles/anthropic-multi-agent-research-system-2025)

## Token Economics & Cost Multipliers

Running parallel agent swarms introduces significant token multiplication:

- Interactive chat is the lowest-cost baseline.
- Autonomous loops consume more context through repeated observation and action.
- Parallel subagents multiply token use again; the additional breadth must justify that cost. [[source: agentic-engineering-trends-2026-synthesis]](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

### Economic Reality Check
The productivity gains must justify the token cost multiplication. Multi-agent patterns work economically when:
1. Independent subtasks can run concurrently.
2. The answer benefits from separate context windows.
3. Quality improvements from adversarial verification justify the overhead.

## Landmark Case Study: FastRender Browser Engine

In January 2026, the Cursor engineering team tested long-running autonomous swarms by building a web browser engine from scratch in Rust (`fastrender`):

- **Scale:** A very large generated Rust codebase spanning many files
- **Architecture:** Hierarchical tree of planners breaking browser specifications into modular tickets  
- **Execution:** Hundreds of concurrent agents over one week across isolated workspaces
- **Verification:** Continuous web conformance test suites with terminal Judge agent
- **Result:** Successfully rendered complex real-world pages (google.com, personal blogs) directly to pixels [[source: agentic-engineering-trends-2026-synthesis]](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

## Orchestration Tooling (2026)

1. **Melty Labs Conductor:** Multiple Claude Code agents in parallel on isolated Git worktrees
2. **Claude Squad:** Multiplexes coding agents across concurrent terminal (`tmux`) panes
3. **Pi Agent Framework:** Lightweight, headless agent engine with offline execution and specialized benchmark plugins [[source: agentic-engineering-trends-2026-synthesis]](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

## Failure Modes

| Symptom | Root cause | Fix |
|---|---|---|
| Agents duplicate work | No task partition before dispatch | Partition by ownership boundaries first ([agents vs workflows](/wiki/agents-vs-workflows)) |
| Context explode per agent | Full history broadcast to all | Give each agent only its slice + a summary pointer |
| Merge conflicts on recombination | Concurrent writes to shared state | Serialize writes through one coordinator or partition the state |

## Rule of Thumb

Add an agent only when the context split saves more tokens than the
coordination costs.

## Related Concepts
- [agent harness engineering](/wiki/agent-harness-engineering)
- [subagents and context management](/wiki/subagents-and-context-management)
- [context engineering](/wiki/context-engineering)
- [automated eval engineering](/wiki/automated-eval-engineering)
- [conformance suites as fitness functions](/wiki/conformance-suites-as-fitness-functions)

---

## Onion Architecture
- URL: https://pyweb.dev/wiki/onion-architecture
- Raw Markdown: https://pyweb.dev/wiki/onion-architecture.md
- Type: concept
- Tags: principle, workflow

# Onion Architecture

An architectural pattern created by [jeffrey palermo](/wiki/jeffrey-palermo) that organizes software into concentric layers with dependencies pointing inward toward the core domain. This pattern directly influenced [clean architecture](/wiki/clean-architecture)'s layered approach and emphasis on the [dependency rule](/wiki/dependency-rule).

## Core Principle

Like an onion's layers, the architecture consists of concentric circles where:
- **Inner layers contain policies** — business rules and domain logic
- **Outer layers contain mechanisms** — infrastructure and implementation details  
- **Dependencies point inward** — outer layers depend on inner layers, never vice versa

## Influence on Clean Architecture

[robert c martin](/wiki/robert-c-martin)'s [clean architecture](/wiki/clean-architecture) synthesis adopted the onion pattern's:
- **Concentric organization** — the four-circle clean architecture structure
- **Inward dependencies** — became the core [dependency rule](/wiki/dependency-rule)
- **Domain protection** — business rules isolated from external changes

The onion pattern established the conceptual foundation for dependency inversion in layered architectures.

 [[source: uncle-bob-clean-architecture-2012]](/wiki/raw/articles/uncle-bob-clean-architecture-2012)

---

## Progressive Disclosure
- URL: https://pyweb.dev/wiki/progressive-disclosure
- Raw Markdown: https://pyweb.dev/wiki/progressive-disclosure.md
- Type: concept
- Summary: Providing context and tooling to AI agents in staged layers on-demand rather than up front.
- Tags: agents, context-engineering, skills

# Progressive Disclosure

**Progressive Disclosure** is an architectural pattern where an agent system loads lightweight pointers or indexes first, and only retrieves full documentation, schemas, or tool definitions when a specific trigger condition is met.


```text
L1  name + description          (always loaded)
L2  SKILL.md body               (loaded on use)
L3  supporting files/scripts    (loaded on explicit need)
```

## The Problem It Solves
If an agent has 100 available skills or API endpoints, injecting all 100 skill bodies into the root prompt consumes tens of thousands of tokens on every turn, risking [context rot](/wiki/context-rot) and [prompt bloat](/wiki/prompt-bloat).

## Three Layers of Disclosure
1. **Tier 1: The Catalog Index:** A compact list of names and 50-character descriptions loaded into the initial context.
2. **Tier 2: The Actionable Manifest:** When an intent matches, the agent invokes a discovery tool (like `skill_view` or `tool_describe`) to load the full contract.
3. **Tier 3: The Deep Primary Source:** The agent inspects supporting scripts, reference docs, or API specs only while executing that sub-phase.

## Benefits
- Drastically lowers baseline token consumption per turn.
- Prevents attention dilution across irrelevant tools.
- Allows agent systems to scale to hundreds of modular skills without degrading reasoning quality.
- Complements [editorial diagrams and visual explanations](/wiki/editorial-diagrams-and-visual-explanations) by routing high-level structural overviews first before loading deep implementation schemas.

## Failure Modes

| Symptom | Root cause | Fix |
|---|---|---|
| Everything loaded up front | Context treated as free | Load levels on demand: metadata always, body on use, files on need (Skill L1/L2/L3 pattern) |
| Critical info hidden too deep | Disclosure without escape hatch | Keep a visible index of what exists so agents know what to request |
| Progressive = fragmented | Levels cut across the wrong seam | Split at natural usage boundaries, not arbitrary file sizes |

## Rule of Thumb

Default to the smallest context that answers the current step; every
always-loaded token must justify itself on every request.

## Related
[prompt bloat](/wiki/prompt-bloat), [context engineering](/wiki/context-engineering), [agents md spec](/wiki/agents-md-spec), [smart zone](/wiki/smart-zone).

---

## Prompt Bloat
- URL: https://pyweb.dev/wiki/prompt-bloat
- Raw Markdown: https://pyweb.dev/wiki/prompt-bloat.md
- Type: concept
- Summary: The accumulation of redundant, generic, or conflicting instructions in system prompts that degrades agent performance.
- Tags: agents, context-engineering, anti-patterns

# Prompt Bloat

**Prompt Bloat** is an anti-pattern where developers pack dozens of disparate instructions, generic style guides, defensive guardrails, and full API references into an LLM's system prompt.

## Symptoms of Prompt Bloat
- **Instruction Shadowing:** Long rule lists cause newer rules to override or contradict earlier directives.
- **High Per-Turn Latency & Cost:** Every single turn re-processes hundreds of bloated prompt tokens before reading the user's message.
- **Sycophancy & Hedging:** Models weighed down by overly cautious instructions spend more time explaining why they can't do something than executing the code.

## How to Kill Prompt Bloat
1. **Apply the Defensibility Test:** For every sentence in a prompt or instruction file, ask: *"Can I defend why this is here with a real failure case?"* If not, delete it.
2. **Move Docs to the Filesystem:** Keep system prompts minimal and point to local files (`docs/`, `skills/`, `CONTEXT.md`) that the agent loads only when needed.
3. **Use Tool Definitions:** Replace large textual output formats with structured tool calling schemas (e.g. JSON schema parameters).

## Related
[context rot](/wiki/context-rot), [progressive disclosure](/wiki/progressive-disclosure), [agents md spec](/wiki/agents-md-spec), [context engineering](/wiki/context-engineering).

---

## Red/Green TDD
- URL: https://pyweb.dev/wiki/red-green-tdd
- Raw Markdown: https://pyweb.dev/wiki/red-green-tdd.md
- Type: concept
- Summary: Test-first development where failing tests are observed before writing minimal implementation code - the core discipline for coding agents.
- Tags: tdd, workflow, agents, coding-guidelines

# Red/Green TDD

Red/Green Test-Driven Development is the ideal engineering discipline for
coding agents. [[source: simon-willison-red-green-tdd-2026]](/wiki/raw/articles/simon-willison-red-green-tdd-2026) Prompting
an agent with "Use red/green TDD" encapsulates a complete loop: write the
test first, observe it fail (RED), write minimal code to make it pass
(GREEN), refactor. It is the human-side pattern behind
[eval driven development](/wiki/eval-driven-development) and [tdd with agents](/wiki/tdd-with-agents).

## The Loop

```text
1. RED    - write a failing test that encodes the next behavior
2. VERIFY - run it; confirm it fails for the expected reason
3. GREEN  - write the minimal implementation that passes
4. REFACTOR - clean up with tests green
```

## Why it is Essential for Agents

1. **Prevents tautological tests:** skipping the RED phase risks writing
   tests that pass trivially without exercising new logic.
  [[source: simon-willison-red-green-tdd-2026]](/wiki/raw/articles/simon-willison-red-green-tdd-2026)
2. **Bounds agent scope:** the failing test defines exact acceptance
   criteria, preventing speculative features or unrequested complexity.
3. **Regression safety:** autonomous modifications without a regression
   suite rapidly break peripheral functionality as the codebase grows.
4. **Cheap verification oracle:** a test suite is a deterministic fitness
   function - the same property that makes [conformance suites as fitness functions](/wiki/conformance-suites-as-fitness-functions)
   work for autonomous optimization loops.

## Failure Modes

| Symptom | Root cause | Fix |
|---|---|---|
| Test passes on first run | Test written after the code (tautology) | Delete it; write the test FIRST against stubbed behavior |
| Agent "fixes" the test instead of the code | Goal inversion under pressure | Rule: implementation changes only until GREEN; test changes need explicit human approval |
| Suite slows to minutes per run | Integration-heavy tests, no unit layer | Keep the RED/GREEN inner loop unit-fast; push integration to a separate suite |
| Red phase skipped under time pressure | Agent treats tests as deliverable, not oracle | Prompt the loop ("confirm the test fails for the expected reason") not just the style name |

## Rule of Thumb

If the agent cannot state what the next failing test asserts in one
sentence, it is not ready to write code. One behavior, one test, one GREEN.

## Related

- [simon willison](/wiki/simon-willison) - source of the agentic-engineering pattern
- [tdd with agents](/wiki/tdd-with-agents) - broader agent TDD workflow
- [eval driven development](/wiki/eval-driven-development) - TDD generalised to probabilistic systems
- [conformance suites as fitness functions](/wiki/conformance-suites-as-fitness-functions) - tests as optimization oracles
- [agentic manual testing](/wiki/agentic-manual-testing), [karpathy four guidelines](/wiki/karpathy-four-guidelines), [agentic engineering patterns](/wiki/agentic-engineering-patterns)

---

## Releasable Patch Rate
- URL: https://pyweb.dev/wiki/releasable-patch-rate
- Raw Markdown: https://pyweb.dev/wiki/releasable-patch-rate.md
- Type: concept
- Tags: agentic-patterns, code-quality, workflow

# Releasable Patch Rate

The **Releasable Patch Rate** is the north-star factory metric for autonomous and AI-assisted software engineering. It measures the fraction of generated changes that satisfy every necessary quality, security, architectural, and operational gate required for production release.

```mermaid
flowchart TD
    G[Candidate Patch] --> C1{Correctness}
    C1 -->|Pass| C2{Regression Safety}
    C1 -->|Fail| F[Rejected / Rework]
    C2 -->|Pass| C3{Security Baseline}
    C2 -->|Fail| F
    C3 -->|Pass| C4{Architecture Conformance}
    C3 -->|Fail| F
    C4 -->|Pass| C5{Scope Authorization}
    C4 -->|Fail| F
    C5 -->|Pass| R[Releasable Patch]
```

## The Multiplicative Reliability Law

Quality is not a scalar average; it is a vector of orthogonal constraints that multiply at the release boundary:

$$\text{Releasable Patch Rate} = P(\text{Correctness} \land \text{Regression Safety} \land \text{Security} \land \text{Architecture} \land \text{Scope})$$

A patch that achieves 100% test coverage and passes linting but leaks an API credential or exceeds its authorized file allowlist is **0% releasable**. Averaging metrics hides catastrophic tail risks.

## High-Leverage Metrics vs. Vanity Proxies

| Dimension | Reliable Metric | Distrusted / Poison Proxy |
|---|---|---|
| **Factory Throughput** | Releasable patches per dollar / hour | Lines of Code (LOC) generated |
| **Delivery Success** | Escaped defect rate & rollback frequency | Merge rate & PR count |
| **Review Efficiency** | Actionable findings per reviewer minute | Review comment count |
| **Verification Strength** | Mutation score & differential assertion passes | Raw line coverage percentage |
| **Scope Discipline** | Touched paths within declared allowlist | "Task Complete" self-report |

## Mitigating the Orchestration Tax

When raw code generation outpaces downstream human review and verification bandwidth, organizations experience the **orchestration tax**. Increasing agent concurrency without scaling independent oracles reduces overall software delivery throughput and stability.

---

## Related Concepts
- [agentic code quality](/wiki/agentic-code-quality) — The risk-conditioned control architecture.
- [agentic software factory](/wiki/agentic-software-factory) — Operating models for autonomous development at scale.
- [five debts of agentic engineering](/wiki/five-debts-of-agentic-engineering) — Debt categories that reduce patch releasability.
- [conformance suites as fitness functions](/wiki/conformance-suites-as-fitness-functions) — Deterministic fitness functions for autonomous loops.

---

## Site-as-Eval-Subject Grading
- URL: https://pyweb.dev/wiki/site-as-eval-subject-grading
- Raw Markdown: https://pyweb.dev/wiki/site-as-eval-subject-grading.md
- Type: concept
- Summary: Grading a static site with deterministic graders and LLM judges, then hill-climbing the design against the scorecard.
- Tags: evaluation, agents, technique, workflow

# Site-as-Eval-Subject Grading

Treating a website (or any shippable artifact) as the **subject of its own
eval suite**: deterministic graders + LLM judges produce a per-column
scorecard, and design/content changes are accepted only when they improve the
scorecard. Generalizes [eval driven development](/wiki/eval-driven-development) from prompts to shipped
interfaces. Grounded in [conformance suites as fitness functions](/wiki/conformance-suites-as-fitness-functions): ambiguous
"make the site better" becomes a fitness function the change must clear.

Built first as the pyweb.dev grader suite (2026-08-30, PR #107): 6 code
graders + 5 planned judges, per-column regression gate.

## Architecture

- **Grader interface** (modeled on Anthropic's cwc-workshops
  eval-driven-agent-development, Apache-2.0): `Grader{name, kind: "code" |
  "judge", description, grade(ctx), pass?(v), scale{min,max,good}}`. Adding a
  metric = appending one object.
- **Shared GraderContext** built once per page (markdown, built HTML,
  frontmatter, graph degree, screenshot). Graders stay pure functions of the
  context.
- **Independent columns, no blended composite** — per [agentic code quality](/wiki/agentic-code-quality):
  quality has independent acceptance dimensions; a single number hides
  regressions. The merge gate is per-column: no regressions, targeted column
  improves.
- **Fixtures frozen per climb** — the page set + judge settings (temp 0,
  pinned seed) are immutable for the duration of an optimization loop;
  changing them restarts the baseline.

## First-baseline results (pyweb.dev, 2026-08-30)

| Column | Mean | Pass rate |
|---|---|---|
| Frontmatter complete | 0.81 | 81% |
| Code-block density | 0.41 | 41% |
| Failure modes | 0.47 | 47% |
| Graph health | 7.96 | 94% |
| Slop words (mech) | 0.03 | 100% |
| Dead links | 0.00 | 100% |

The baseline immediately drove one content fix (missing frontmatter on 19%
of concepts) and one framework fix (below).

## Discoveries from building it

### 1. Machine-readable ground truth lives where the build puts it, not where you expect
The wiki edge graph is written to `dist/api/graph.json` by the build step —
NOT `public/api/graph.json`. A grader that reads the wrong path silently
degrades to zero (empty fallback) and reports **0% graph health** — a wrong
number that looks like a real finding. **Rule: before trusting any
deterministic grader's first run, manually verify one non-trivial value by
hand.** Zero-feeling results deserve suspicion in both directions.

### 2. Mechanical slop lists have false positives on technical text
"seamlessly" flagged a sentence describing an adversarial pipeline switching
providers — accurate technical description, not slop. "holistic" flagged a
legitimate diagnosis of model limitations. Both single-word flags are
**context-dependent**; the mechanical column is a cheap negative filter (its
pass = "no slop") but flags must be human-reviewed before becoming content
edits. Pattern-level judges (no-ai-slop style: "binary contrasts",
"throat-clearing openers") have far lower false-positive rates than
word-level lists.

### 3. The Goodhart surface is per-column
Each grader is independently gameable: stuffing code blocks games
code-density; shoehorning "failure" mentions games failure-modes; adding
wikilinks games graph health. Mitigations: (a) an objective-adherence judge
that checks whether changes serve the actual reader goal; (b) per-column
regression gate so gaming one column cannot hide damage in another; (c)
calibration against human judgment before trusting judge columns.

### 4. Independent verification catches what self-report cannot
Adversarial audit of the first smoke report found the slop false positives
within minutes — the framework author (the agent) had classified them as
100% pass without reading the flagged contexts. **An eval framework built by
an agent needs a second agent (or human) to adversarially audit it before
its numbers are trusted.** [designing for verifiability](/wiki/designing-for-verifiability) applies to the
grader itself: report metric definitions so an auditor can recompute.

## Judge best practices (from OpenAI guidance + Hamel Husain + web research)

- **Anchored exemplars**: include 2-3 canonical snapshots in the judge prompt
  at fixed scores (a 2/5 "poor hierarchy" example, a 5/5 gold standard) so
  the judge calibrates against references, not vibes.
- CoT-before-score; structured outputs (zod strict) for machine-grading;
  numeric score AND pass/fail threshold per column.
- Pairwise comparison with position order swapped twice cancels position
  bias when judging a redesign against the current site.
- Judge noise check: two runs of the same fixture must agree within
  tolerance before any climb starts.

### Adopted from independent web research (2026-08-30)

1. **Mutation testing of judges** (OpenAI): deliberately inject defects
   (broken links, low-contrast headings, malformed metadata) and assert each
   judge's score drops. An insensitive judge prompt is rewritten, not
   trusted.
2. **Multi-persona panel**: decompose design judging into orthogonal
   personas (information architect, first-time reader, accessibility
   auditor) aggregated by weighted consensus, reducing bias toward generic
   aesthetic tropes.
3. **Holdout split**: hill-climb against a training subset of pages; block
   release if the holdout set fails to generalize. Generalization beats
   fixture-fit.
4. **Grounded evidence locators**: judges must cite exact text snippets / DOM
   selectors / bounding boxes for every deduction; deductions without
   evidence are discarded.
5. **Discrete coarse rubrics** (1-4 with strict per-step criteria, not
   1-100 floats) + N=3 majority voting on borderline scores to fight
   non-determinism on edge cases.
6. **Known failure modes to watch**: LLM aesthetic homogenization (all
   judged sites converge to generic AI style) - separate stylistic grading
   from structural utility; viewport truncation blindspots - pair full-page
   screenshots with DOM/accessibility dumps.

### Novel wiki ideas harvested

- **Dual-stream multimodal verification**: computed DOM accessibility trees
  + multi-viewport rendered snapshots produce hallucination-proof site
  judges.
- **Evolutionary anti-regression invariant mining**: automatically promote
  repeated judge findings into deterministic code graders, cutting judge
  token cost over time (e.g., after the slop judge flags a pattern three
  runs, add it to the mechanical slop-words list).

## Related

[eval taxonomy](/wiki/eval-taxonomy) (three tiers), [conformance suites as fitness functions](/wiki/conformance-suites-as-fitness-functions)
(the climb loop), [agentic code quality](/wiki/agentic-code-quality) (independent dimensions),
[eval driven development](/wiki/eval-driven-development) (error analysis first),
[designing for verifiability](/wiki/designing-for-verifiability), [evals skills](/wiki/evals-skills) (graders as skills),
[generator evaluator loop](/wiki/generator-evaluator-loop).

---

## Skill Treatment Effect
- URL: https://pyweb.dev/wiki/skill-treatment-effect
- Raw Markdown: https://pyweb.dev/wiki/skill-treatment-effect.md
- Type: concept
- Tags: agentic-patterns, evaluation, harness-engineering

# Skill Treatment Effect

The **Skill Treatment Effect** refers to the measured marginal utility of injecting procedural knowledge packages ([evals skills](/wiki/evals-skills), `SKILL.md`) into an agent's context during inference. Empirical benchmarks demonstrate that skills are not a universal productivity booster; their effectiveness depends heavily on domain specificity, task fit, and version compatibility.

```mermaid
flowchart LR
    A[Raw Skill Injection] --> B{Task & Domain Fit}
    B -->|Domain-Specific / Tool Procedure| C[+15% to +30% Lift]
    B -->|Generic Software Engineering| D[+1.2% to +4.5% Marginal Lift]
    B -->|Version Mismatch / Contradiction| E[-5% to -10% Regression & Bloat]
```

## Empirical Evidence

### SkillsBench (Cross-Domain Benchmark)
- **Scope:** 87 tasks across 8 domains evaluated over 18 model-harness configurations.
- **Aggregate Result:** Curated skills increased the average pass rate from 33.9% to 50.5% (+16.6 percentage points; 25.5% normalized gain).
- **Domain Variance:** Broad non-coding tasks saw substantial lift, but software engineering tasks experienced only a modest **+4.5 pp** improvement.
- **Composition Law:** Focused skills with $\le 3$ modules outperformed exhaustive or kitchen-sink bundles. Self-generated skills yielded zero net benefit.

### SWE-Skills-Bench (Software Engineering Focus)
- **Scope:** 49 public SWE skills across ~565 task instances in real-world GitHub repositories with deterministic verifiers.
- **Aggregate Result:** Mean pass-rate gain across all skills was only **+1.2%**.
- **Distribution:**
  - **39 of 49 skills:** Zero pass-rate improvement while increasing token overhead by up to 451%.
  - **7 specialized skills:** Delivered meaningful gains (up to +30%) by providing toolchain-specific API knowledge or complex domain flows.
  - **3 mismatched skills:** Degraded performance (up to -10%) due to outdated assumptions or version incompatibilities conflicting with the repo.

## Practical Implications for Harness Engineering

1. **Skills Are Code Interventions:** Treat skills like code dependencies. Version them, evaluate them on paired benchmark tasks (with vs. without skill), and track token cost vs. pass-rate delta.
2. **Beware Prompt Bloat:** Generic advice (e.g., "write clean modular code") induces [prompt bloat](/wiki/prompt-bloat) and increases inference latency without moving verification needles.
3. **Progressive Disclosure:** Pre-load only skill names and triggers in the root prompt; load the full procedural body only when the task explicitly routes to that skill.

---

## Related Concepts
- [evals skills](/wiki/evals-skills) — Evaluating and testing agent skills.
- [agent harness engineering](/wiki/agent-harness-engineering) — Harness architecture and runtime orchestration.
- [prompt bloat](/wiki/prompt-bloat) — Degradation of performance from excessive prompt instructions.
- [progressive disclosure](/wiki/progressive-disclosure) — Staged context loading to avoid token saturation.

---

## Smart Zone
- URL: https://pyweb.dev/wiki/smart-zone
- Raw Markdown: https://pyweb.dev/wiki/smart-zone.md
- Type: concept
- Summary: The optimal token window where frontier LLMs maintain peak reasoning sharpness before attention degradation sets in.
- Tags: agents, context-engineering, workflow

# Smart Zone

The **Smart Zone** refers to the initial token budget of a session (typically between 0 and 150k tokens on modern frontier models) during which an AI agent operates at maximum reasoning sharpness, exactness, and adherence to complex instructions.

## Why the Smart Zone Exists
Although modern models advertise 1M+ or 2M+ token context windows, their effective reasoning capacity is not constant throughout the entire window.

As a conversation approaches 100k–150k tokens:
- Latency and compute costs scale non-linearly.
- Subtle constraints in the system prompt are more frequently forgotten.
- Refactoring operations become sloppy and prone to regressions.

## Operating in the Smart Zone
1. **Bounded Phases:** Keep each phase (grilling, spec, implementation) within the smart zone.
2. **Phase Boundary Compaction:** At the boundary of a phase (e.g. going from specification to implementation), compress state into a [handoff artifact](/wiki/handoff-artifacts) and start a fresh session context.
3. **One Ticket Per Fresh Context:** Run each implementation ticket in a clean context window rather than daisy-chaining multiple major feature builds in one marathon session.

## Related
[context rot](/wiki/context-rot), [handoff artifacts](/wiki/handoff-artifacts), [context engineering](/wiki/context-engineering), [subagents and context management](/wiki/subagents-and-context-management), [ai coding taxonomy](/wiki/ai-coding-taxonomy).

---

## Software Engineering Fundamentals for Agents
- URL: https://pyweb.dev/wiki/software-engineering-fundamentals-for-agents
- Raw Markdown: https://pyweb.dev/wiki/software-engineering-fundamentals-for-agents.md
- Type: concept
- Tags: agentic-patterns, architecture, workflow, code-quality

# Software Engineering Fundamentals for Agents

Software engineering fundamentals for agents is an architectural discipline and steering framework formulated by [andrew ng](/wiki/andrew-ng). It posits that while coding agents make syntax generation virtually free, deep comprehension of core software fundamentals is the decisive capability required to steer agents across non-negotiable engineering tradeoffs. [[source: andrew-ng-software-engineering-fundamentals-2026]](/wiki/raw/articles/andrew-ng-software-engineering-fundamentals-2026)

Without these fundamentals, unguided or "vibe-coding" approaches cause agents to make catastrophic tradeoffs in latency, availability, consistency, reliability, maintainability, simplicity, and operational cost.

```mermaid
flowchart TD
    subgraph Human Steering & Constraints
        F1[Full-Stack Mechanics]
        F2[Data Architecture]
        F3[System Decomposition]
        F4[Reliability & Security]
        F5[Production Operations]
    end

    subgraph Agent Execution Harness
        F1 -->|API & State Boundaries| AG[Coding Agent Generator]
        F2 -->|Storage & Access Patterns| AG
        F3 -->|Modular Seams & Granularity| AG
        F4 -->|Verification Suites & Blast Radius| AG
        F5 -->|CI/CD & Observability| AG
    end

    AG --> RES[Durable, Scalable Production System]
```

## The Five Essential Pillars

### 1. Full-Stack Application Mechanics
- **Agent Amplification:** Enables specialized developers (e.g. mobile or frontend engineers) to function as full-stack engineers by generating code outside their primary specialization.
- **Steering Requirements:** The engineer must understand frontend and backend interaction models: page rendering paradigms (SSR, SSG, client hydration), caching hierarchies, API protocol selection (REST, GraphQL, gRPC), authentication flows, state/session distribution, asynchronous worker pipelines, and accessibility standards.

### 2. Data Architecture & Lifecycle Management
- **The Data Lock-in Problem:** Data models form the durable substrate of software. Schema and persistence mistakes are expensive to migrate even with agent assistance.
- **Steering Requirements:** Designing appropriate access patterns, selecting storage paradigms (relational, document, key-value, graph), and handling transactional concurrency, data cleanliness, and privacy/governance.
- **Agent Context Feed:** AI applications ingest domain context directly from data infrastructure; poorly architected data layers starve agents of the context required to make sound decisions ("the AI doesn't know what it doesn't know").

### 3. Evolutionary System Architecture
- **Dynamic Boundaries:** Architecture is a moving target across project phases (throwaway prototype $\to$ initial production $\to$ scaled deployment).
- **Steering Requirements:** Decomposing systems cleanly, setting explicit boundaries between client and server, placing application state, choosing architectural granularity (monolith vs. microservices), and running targeted experiments before locking in dependencies.

### 4. Reliability & Shift-Left Security
- **Verification Strategy:** Defining structured verification mixes (unit, integration, end-to-end) and coverage criteria rather than trusting unchecked generator output.
- **Failure Containment:** Designing graceful degradation, circuit breaking, rate limit backoff, and strict containment to minimize failure blast radius.
- **Shift-Left Security:** Moving vulnerability scanning, dependency supply-chain auditing, and attack-surface analysis into the early development loop.

### 5. Production Operations & Scaling
- **SDLC & Delivery:** Mastering automated deployment pipelines (CI/CD), environment isolation, and Infrastructure as a Service (IaaS).
- **Operational Health:** Establishing real-time observability (structured telemetry, metrics, distributed tracing, alerting, and incident response).
- **Elastic Scaling:** Load balancing, horizontal vs. vertical scaling, database partitioning/sharding, and ongoing technical debt management.

---

## Comparison: Vibe Coding vs. Grounded Agentic Engineering

| Dimension | Vibe Coding (Unsteered) | Grounded Agentic Engineering (Steered) |
| :--- | :--- | :--- |
| **Primary Focus** | Syntax generation & immediate UI behavior | System invariants, contracts, and boundary enforcement |
| **Data Layer** | Ad-hoc schemas and unstructured persistence | Access-pattern optimized storage with clear lifecycle rules |
| **Failure Handling** | Fragile catch-all blocks; undetected regressions | Explicit blast radius containment & automated verification |
| **Architecture** | Accidental complexity; tangled monolithic coupling | Deliberate seams, clean interfaces, and evolutionary paths |
| **Security** | Opaque third-party deps and unchecked endpoints | Shift-left scanning, least-privilege sandboxing, and audit trails |

---

## Related
- [ai engineer role](/wiki/ai-engineer-role) — the role definition that motivates these fundamentals. Concepts
- [agentic engineering patterns](/wiki/agentic-engineering-patterns) — Disciplined verification loops (Simon Willison).
- [five debts of agentic engineering](/wiki/five-debts-of-agentic-engineering) — The structural failure modes of unguided agentic code.
- [agent containment and blast radius](/wiki/agent-containment-and-blast-radius) — Restricting agent execution scope and operational risk.
- [clean architecture](/wiki/clean-architecture) — Maintaining strict modular boundaries across evolving systems.
- [context engineering](/wiki/context-engineering) — Structuring the information environment provided to AI models.

---

## Structured Outputs
- URL: https://pyweb.dev/wiki/structured-outputs
- Raw Markdown: https://pyweb.dev/wiki/structured-outputs.md
- Type: concept
- Summary: Constraining LLM responses to JSON-schema shapes — objects, enums, arrays — for data extraction and classification, with streaming and tool-based variants.
- Tags: llm-fundamentals, context-engineering, patterns

# Structured Outputs

Often the thing you want back from an LLM is **not text but an object**: extracting multiple properties from unstructured input (PDFs, comments, documents) is one of the most powerful and business-relevant LLM use cases, alongside classification into enums/categories.


```json
{
  "name": "verdict",
  "strict": true,
  "schema": {
    "type": "object",
    "properties": { "pass": {"type": "boolean"}, "reason": {"type": "string"} },
    "required": ["pass", "reason"],
    "additionalProperties": false
  }
}
```

## Variants
- **Object generation:** pass a JSON schema, receive a typed object (data extraction from PDFs and other unstructured sources).
- **Enum generation:** constrain output to a fixed set of enumerated values (classification, sentiment, routing).
- **Array generation:** schema with multiple items for batch extraction.
- **Streaming objects:** instead of waiting for the whole object, stream it field-by-field as generation proceeds.
- **Tool-based structuring:** reuse [tool calling loop](/wiki/tool-calling-loop) infrastructure — declare a tool whose arguments are your schema and force the model to call it — to get the same shape guarantees without a dedicated structured-output API.

## Why It Matters
Structured outputs are the bridge between probabilistic text generation and deterministic application code: they make LLM output consumable by databases, validators, and pipelines. The pattern pairs naturally with [evals skills](/wiki/evals-skills) — schema conformance is cheap to verify automatically, a rare machine-checkable oracle.

## Failure Modes

| Symptom | Root cause | Fix |
|---|---|---|
| Schema violated at runtime | Model improvises fields | Strict schema mode + validation retry on parse failure |
| Over-constrained schema chokes output | Too many required fields | Required only what downstream code reads; everything else optional |
| Silent parse fallback | JSON extracted with regex | Fail loudly on malformed output; route to retry, never to guess |

## Rule of Thumb

Schema conformance is the cheapest deterministic oracle available -
prefer it over any judge when the property is checkable.

## Related
[llm message protocol](/wiki/llm-message-protocol), [tool calling loop](/wiki/tool-calling-loop), [model provider abstraction](/wiki/model-provider-abstraction), [evals skills](/wiki/evals-skills), [generator evaluator loop](/wiki/generator-evaluator-loop).

---

## Subagents and Context Management
- URL: https://pyweb.dev/wiki/subagents-and-context-management
- Raw Markdown: https://pyweb.dev/wiki/subagents-and-context-management.md
- Type: concept
- Summary: Preserving root conversation context by dispatching isolated sub-agents for repo exploration, testing, debugging, and file edits.
- Tags: agents, subagents, context-engineering, workflow

# Subagents and Context Management

LLM context windows carry token budgets and degrade in reasoning sharpness as conversations grow bloated. Subagents solve this by isolating high-token exploratory work in ephemeral child contexts, returning only concise summaries to the root agent.


```text
dispatch  -> subagent context = goal + minimal files + output schema
collect   -> parent receives result summary only (not the transcript)
```

## Core Roles
- **Exploration (e.g. Claude Code Explore):** An isolated subagent searches the tree, locates relevant views/templates/functions, and hands back exact file paths and line numbers without flooding the root session.
- **Parallel Subagents:** Independent tasks (e.g. updating 5 decoupled template files or refactoring independent modules) dispatched concurrently across child agents to accelerate throughput.
- **Specialist Subagents:** Delegating specific roles like code reviewer, verbose test-runner filter, or debugger to dedicated prompts with customized system instructions.

## Context Preservation Rule
The primary value of subagents is not unnecessary complexity, but preserving the root agent's context window for architectural decisions, high-level verification, and user steering.

## Failure Modes

| Symptom | Root cause | Fix |
|---|---|---|
| Subagent returns useless summary | Goal under-specified at dispatch | Goal = task + files + output schema, written down before dispatch |
| Parent context explodes on collect | Full transcripts merged back | Collect result summaries only; transcripts stay in the subagent |
| Subagents duplicate work | Overlapping scopes | Partition by file/area ownership before dispatch |
| Lost consensus across agents | No synthesis step | One parent-side synthesis pass over all summaries |

## Rule of Thumb

## Rule of Thumb

A subagent gets a goal, the minimum files, and an output schema - never
the parent transcript. If it needs the transcript, the task split is
wrong.

## Related
- [simon willison](/wiki/simon-willison)
- [agentic engineering patterns](/wiki/agentic-engineering-patterns)
- [context engineering](/wiki/context-engineering)
- [karpathy four guidelines](/wiki/karpathy-four-guidelines)

---

## TDD With Agents
- URL: https://pyweb.dev/wiki/tdd-with-agents
- Raw Markdown: https://pyweb.dev/wiki/tdd-with-agents.md
- Type: concept
- Summary: Rigorous test-driven development loop using executable tests as the truth referee for coding agents.
- Tags: tdd, workflow, agents, technique

# TDD With Agents

Test-driven development as the control mechanism for agent-produced code. Write the
test first (RED), make it pass (GREEN), then refactor. With agents, the test is the
verifiable success criteria that lets the agent loop independently.

## Why it matters now
[matt pocock](/wiki/matt-pocock): "Bad code is now the most expensive it has ever been." Agents ship volume.
Without tests as the gate, the volume is slop. TDD turns [karpathy four guidelines](/wiki/karpathy-four-guidelines) #4
(goal-driven execution) from principle into practice: "fix the bug" → "write a test that
reproduces it, then make it pass." See also [red green tdd](/wiki/red-green-tdd) and [simon willison](/wiki/simon-willison)'s [agentic engineering patterns](/wiki/agentic-engineering-patterns).

## The loop
1. RED — write a failing test that encodes the requirement.
2. GREEN — write minimum code to pass the test. Nothing speculative.
3. REFACTOR — improve structure without changing behavior. [karpathy four guidelines](/wiki/karpathy-four-guidelines)
   #3 (surgical changes) governs this step.

## How agents change it
- The test IS the spec for the agent. A good test lets the agent loop autonomously
  because it knows when it's done.
- Each [tracer bullet](/wiki/tracer-bullets) ticket should ship with its test — the slice is
  verifiable because the test exists.
- The agent can run the test and report real results — no "I think it works."

## Apply it (Agent Recipe)
- **Role:** Test-driven implementation specialist.
- **When to trigger:** Writing new functionality or fixing a reported defect.
- **Instruction:**
  1. Write an isolated test case asserting target behavior.
  2. Run the test suite and verify the test fails for the expected reason (RED).
  3. Write the minimal implementation required to pass the test (GREEN).
  4. Refactor code structure while keeping the test suite green (REFACTOR).
- **Verification:**
  ```bash
  npm test
  ```
- **Pitfalls to avoid:** Writing tests after implementation; mocking out the actual unit under test.

## Related
[karpathy four guidelines](/wiki/karpathy-four-guidelines), [tracer bullets](/wiki/tracer-bullets), [idea to ship flow](/wiki/idea-to-ship-flow), [grilling doctrine](/wiki/grilling-doctrine), [red green tdd](/wiki/red-green-tdd).

---

## The Feynman Technique
- URL: https://pyweb.dev/wiki/feynman-technique
- Raw Markdown: https://pyweb.dev/wiki/feynman-technique.md
- Type: concept
- Summary: Teaching and learning methodology: plain English explanations, identifying gaps, and radical simplification.
- Tags: technique, teaching, pedagogy

# The Feynman Technique

Learning and teaching method: understanding is tested by whether you can explain the concept
simply. Named for [richard feynman](/wiki/richard-feynman); his Caltech colleagues called him "the great
explainer".

## The loop
1. Say what you know in PLAIN language, as if to a bright 12-year-old. No jargon until
   the idea lands.
2. Find the gap — where the plain explanation stumbles is exactly where understanding is
   missing. Return to the source for just that piece.
3. Simplify and use analogy. If the explanation needs the technical term to survive, it
   isn't understood yet.
4. Test by teaching: have the learner explain it back or predict an outcome ("what does
   this print?"). Their gap is the next lesson. Iterate.

## Why it works
Jargon can hide a hole; plain language can't. Feynman's Brazilian-physics-students story:
they could recite definitions but not apply them — names without the thing.

## Application to AI Systems & Evals
When evaluating agent understanding and reasoning capabilities, plain-language decomposition
exposes latent gaps in system prompts and test rubrics. Complemented by [build from scratch pedagogy](/wiki/build-from-scratch-pedagogy) for deep technical
topics where building a toy prototype makes understanding executable.

## Related
[first principles thinking](/wiki/first-principles-thinking), [build from scratch pedagogy](/wiki/build-from-scratch-pedagogy), [richard feynman](/wiki/richard-feynman), [andrej karpathy](/wiki/andrej-karpathy).

---

## Think Tool
- URL: https://pyweb.dev/wiki/think-tool
- Raw Markdown: https://pyweb.dev/wiki/think-tool.md
- Type: concept
- Summary: Anthropic's technique of giving an LLM a no-op 'think' tool so it can persist structured reasoning into context before complex tool calls.
- Tags: agents, tool-use, patterns

# Think Tool

Anthropic's think-tool technique: give the LLM a **`think` tool that does nothing** — its `execute` simply returns the `thought` argument passed to it. The value is not an action but a **structured pause**: the thought is appended to the message history, saving important information in context so later iterations of the [tool calling loop](/wiki/tool-calling-loop) can make better decisions.^[raw/aihero/implementing-anthropics-think-tool-in-typescript.md]

```mermaid
flowchart LR
    A["Complex tool call"] --> T["Think (no-op)"] --> B["Next tool call, informed"]
```

## Mechanics
The tool is ordinary tool-calling: a description ("use it when complex reasoning or some cache memory is needed — it will not obtain new information or change the database, just append the thought to the log") plus a JSON-schema `thought: string` parameter. The `execute` function returns the thought unchanged; persistence in the [llm message protocol](/wiki/llm-message-protocol) history does the rest. Echoes ReAct and Reflexion.

## Why It Works
The model gets an explicit, schedulable slot to reflect between steps instead of cramming all reasoning into one response — a tool-surface application of the same principle behind [smart zone](/wiki/smart-zone) and context staging: put the right information in the right place in history for the decision that needs it.

## Failure Modes

| Symptom | Root cause | Fix |
|---|---|---|
| Tool call storms without progress | Agent retries failing calls blindly | Force a think step after every failed call; log the diagnosis |
| Notes ignored on later turns | Scratchpad not re-injected | Re-inject the note summary into subsequent prompts |
| Think used as a delay tactic | No binding between thoughts and next action | Each thought must name the next concrete action |

## Rule of Thumb

One thought per decision point; a thought that does not change the next
action was wasted context.

## Related
[tool calling loop](/wiki/tool-calling-loop), [llm message protocol](/wiki/llm-message-protocol), [generator evaluator loop](/wiki/generator-evaluator-loop), [smart zone](/wiki/smart-zone), [context engineering](/wiki/context-engineering).

---

## Tool Calling Loop
- URL: https://pyweb.dev/wiki/tool-calling-loop
- Raw Markdown: https://pyweb.dev/wiki/tool-calling-loop.md
- Type: concept
- Summary: The four-step agentic loop — specify tools, receive tool calls, execute locally, return results — that lets LLMs act on the world.
- Tags: agents, tool-use, llm-fundamentals

# Tool Calling Loop

Tools (also called functions) are the mechanism that turns an LLM from a text generator into an agent that can **act on the world and get feedback from it**.

## The Loop
```mermaid
sequenceDiagram
    participant App as Application
    participant LLM as LLM
    App->>LLM: System prompt with tool definitions (name, description, JSON-schema args)
    App->>LLM: User message ("write a .gitignore file")
    LLM-->>App: Tool call message (id, tool name, arguments)
    App->>App: Execute tool locally (e.g. write file)
    App->>LLM: Tool result message (id, "wrote .gitignore successfully")
    LLM-->>App: Summary text of what was done
```

1. **Specify** tools in the system prompt: each is a name, a description, and JSON-schema-typed arguments. Nothing more — tool definitions are just extra prompt information.
2. **Call:** the LLM replies with a special tool-call message carrying an ID, the tool name, and the argument object. Nothing has happened yet in the world at this point.
3. **Execute:** the application runs its own code to perform the action.
4. **Result:** the application returns a tool-result message with the matching ID; the LLM follows with a human-facing summary.

A tool call is "really like a conversation with the LLM — it's just the LLM communicating with the system that creates the file instead of communicating with us."

## Tool Budget Caution
Too many tools is actively detrimental: with 40+ tool definitions the context window drowns and selection accuracy drops ([context rot](/wiki/context-rot), lost-in-the-middle). Many frameworks recommend staying **under ~6 tools**; issues can appear as low as 12. Tool-count discipline is therefore a first-class [context engineering](/wiki/context-engineering) concern, not a nice-to-have.

## Failure Modes

| Symptom | Root cause | Fix |
|---|---|---|
| Agent calls tools in wrong order | No dependency awareness | Prompt declares tool preconditions explicitly |
| Infinite retry on failing tool | Error treated as transient | Cap retries; require a diagnosis after 2 failures |
| Hallucinated parameters | Schema ambiguity | Strict schemas; validate before execution |
| >6 tools confuse selection | Tool-choice accuracy degrades with count | Group or gate tools by task phase |

## Related
[llm message protocol](/wiki/llm-message-protocol), [model context protocol basics](/wiki/model-context-protocol-basics), [context rot](/wiki/context-rot), [agent harness engineering](/wiki/agent-harness-engineering), [generator evaluator loop](/wiki/generator-evaluator-loop).

---

## Tracer Bullets
- URL: https://pyweb.dev/wiki/tracer-bullets
- Raw Markdown: https://pyweb.dev/wiki/tracer-bullets.md
- Type: concept
- Summary: Shipping the thinnest end-to-end slice through all architecture layers before expanding features.
- Tags: workflow, agents, technique

# Tracer Bullets

Ship the smallest end-to-end slice first — one thin path through every layer — then thicken.
Term from *The Pragmatic Programmer*; [matt pocock](/wiki/matt-pocock) repurposes it as the main defense
against AI slop: agents produce volume, so constrain each ticket to a verifiable slice. [[source: aihero-dev-homepage-2026]](/wiki/raw/articles/aihero-dev-homepage-2026)


```text
ticket = slice-through-all-layers (UI stub -> data -> back)
verify = demo the slice end-to-end
then   -> thicken, one layer at a time
```

## Why it beats big-bang with agents
- Reality referees early: integration lies surface on slice one, not week three
  ([richard feynman](/wiki/richard-feynman): "nature cannot be fooled" — production either).
- Each slice has crisp success criteria → enables [karpathy four guidelines](/wiki/karpathy-four-guidelines) #4
  goal-driven looping.
- Review load stays human-sized; slop can't hide in a 4,000-line diff.

## In practice
Decompose specifications into tracer-bullet tickets with explicit blocking edges;
each implementation phase builds one slice via [tdd with agents](/wiki/tdd-with-agents). See [idea to ship flow](/wiki/idea-to-ship-flow).
"What I cannot create, I do not understand" — Feynman's dictum; the tracer bullet is the created-therefore-understood proof, per ticket.

## Failure Modes

| Symptom | Root cause | Fix |
|---|---|---|
| Slice never reaches production | "Thin" interpreted as UI-only, skipping data layer | Slice must touch every layer, even with stubs |
| Slices sprawl into mini-projects | No verifiable done-criteria per slice | Each slice gets one observable success check before the next starts |
| Agent gold-plates slice one | No explicit scope fence | Constrain the ticket: "only the slice; do not extend" ([karpathy four guidelines](/wiki/karpathy-four-guidelines)) |

## Rule of Thumb

If a ticket cannot be demoed end-to-end in one sitting, it is not a tracer
bullet - split it. The demo IS the verification.

## Related
[idea to ship flow](/wiki/idea-to-ship-flow), [tdd with agents](/wiki/tdd-with-agents), [karpathy four guidelines](/wiki/karpathy-four-guidelines), [richard feynman](/wiki/richard-feynman).

---


# Section: ENTITIES

## Addy Osmani
- URL: https://pyweb.dev/wiki/addy-osmani
- Raw Markdown: https://pyweb.dev/wiki/addy-osmani.md
- Type: entity
- Summary: Engineering leader and author writing on agent harness engineering, agentic code quality gates, and software factory workflows.
- Tags: person, workflow

# Addy Osmani

Software engineer, engineering leader, and author whose writing establishes core principles of agent harness design and quality engineering in the agentic software era. [[source: agentic-engineering-trends-2026-synthesis]](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

## Core Contributions

### Agent Harness Engineering
Osmani authored foundational writing on agent harness engineering, establishing the distinction between raw model intelligence and the surrounding operating environment, tools, and execution loop. [[source: agentic-engineering-trends-2026-synthesis]](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

### Constraint-Driven Agentic Code Quality
In "Agentic Code Quality" (2026), Osmani articulated that as autonomous agents generate code at machine speeds, manual line-by-line human code review ceases to scale. Software quality becomes a direct function of the deterministic constraints, feedback loops, and back-pressure engineered into the agent harness. [[source: addy-osmani-agentic-code-quality-2026]](/wiki/raw/articles/addy-osmani-agentic-code-quality-2026)

Key tenets:
- **Quality as Constraints:** Harnesses enforce boundaries via multi-tier checks (linters, type-checkers, property-based tests, mutation testing, and architectural dependency rules).
- **Earned Autonomy:** Changes route through tiered autonomy based on risk profile, blast radius, and historical verification track records.
- **Human Attention Economics:** Proactively routing scarce human judgment to architecture, intent, and exceptional failures rather than syntax.

## Cross-links
- [agentic code quality](/wiki/agentic-code-quality) — constraint-driven verification and multi-tier feedback
- [agent harness engineering](/wiki/agent-harness-engineering) — operating systems and harnesses for autonomous agents
- [agentic software factory](/wiki/agentic-software-factory) — operating model for scaled agent delivery
- [simon willison](/wiki/simon-willison) — contemporary in developer tooling and AI workflows
- [viv trivedy](/wiki/viv-trivedy) — harness engineering pioneer

---

## Aditi Raghunathan
- URL: https://pyweb.dev/wiki/aditi-raghunathan
- Raw Markdown: https://pyweb.dev/wiki/aditi-raghunathan.md
- Type: entity
- Tags: person

# Aditi Raghunathan

Aditi Raghunathan is a co-author of *Adversarial Hacker Fixer Verifiers 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/adversarial-hacker-fixer-verifiers-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Ahmed E. Hassan
- URL: https://pyweb.dev/wiki/ahmed-e-hassan
- Raw Markdown: https://pyweb.dev/wiki/ahmed-e-hassan.md
- Type: entity
- Tags: person

# Ahmed E. Hassan

Ahmed E. Hassan is a co-author of *Agentic Pull Requests Github 2026* and *Aidev Dataset 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/agentic-pull-requests-github-2026.md]^[raw/papers/aidev-dataset-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Airbnb
- URL: https://pyweb.dev/wiki/airbnb
- Raw Markdown: https://pyweb.dev/wiki/airbnb.md
- Type: entity
- Summary: Global travel marketplace and engineering pioneer in industrial Eval-Driven Development (EDD) for generative AI applications.
- Tags: organization, evaluation, workflow

# Airbnb

Airbnb is a global accommodation and travel platform whose engineering organisation has pioneered the methodology of [eval driven development](/wiki/eval-driven-development) (EDD) for GenAI systems at scale. [[source: airbnb-eval-driven-development-2026]](/wiki/raw/articles/airbnb-eval-driven-development-2026)

## Contributions to GenAI Engineering

Airbnb established core industrial practices for evaluating generative AI systems, formalizing the **Three-Layer Eval Funnel**:
1. **Layer 1:** Fast, deterministic programmatic checks (schemas, AST, regex).
2. **Layer 2:** Small, sharp LLM-as-a-judge evaluators focused on single orthogonal dimensions.
3. **Layer 3:** High-leverage human evaluation for trace discovery and policy calibration.

Their foundational rule ("When in doubt, look at your data") mandates inspecting 100 baseline execution traces to derive targeted evaluators rather than creating theoretical metrics in isolation.

## Related
- [eval driven development](/wiki/eval-driven-development) — core methodology
- [agentic code quality](/wiki/agentic-code-quality) — multi-tier quality architecture
- [error analysis and evals](/wiki/error-analysis-and-evals) — trace analysis
- [hamel husain](/wiki/hamel-husain) — evals methodology

---

## Alistair Cockburn
- URL: https://pyweb.dev/wiki/alistair-cockburn
- Raw Markdown: https://pyweb.dev/wiki/alistair-cockburn.md
- Type: entity
- Tags: person, educator

# Alistair Cockburn

Software development methodologist and author of [hexagonal architecture](/wiki/hexagonal-architecture) (also known as Ports and Adapters pattern). His architectural pattern influenced [clean architecture](/wiki/clean-architecture) and emphasizes isolating business logic from external concerns through well-defined interfaces.

## Key Contributions

### Hexagonal Architecture
Created the Ports and Adapters architectural pattern that separates core business logic from external systems through explicit port definitions. This pattern was adopted by [steve freeman](/wiki/steve-freeman) and [nat pryce](/wiki/nat-pryce) in their book "Growing Object Oriented Software" and integrated into [robert c martin](/wiki/robert-c-martin)'s [clean architecture](/wiki/clean-architecture) synthesis.

## Related Concepts
- [hexagonal architecture](/wiki/hexagonal-architecture) — his core architectural contribution
- [clean architecture](/wiki/clean-architecture) — incorporates his hexagonal patterns

 [[source: uncle-bob-clean-architecture-2012]](/wiki/raw/articles/uncle-bob-clean-architecture-2012)

---

## Andrei Botocan
- URL: https://pyweb.dev/wiki/andrei-botocan
- Raw Markdown: https://pyweb.dev/wiki/andrei-botocan.md
- Type: entity
- Tags: person

# Andrei Botocan

Andrei Botocan is a co-author of *Autonomous Agent Contributions Wild 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/autonomous-agent-contributions-wild-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Andrej Karpathy
- URL: https://pyweb.dev/wiki/andrej-karpathy
- Raw Markdown: https://pyweb.dev/wiki/andrej-karpathy.md
- Type: entity
- Summary: AI researcher, educator, Stanford CS231n co-creator, and developer of the build-from-scratch pedagogy.
- Tags: person, educator, agents, pedagogy

# Andrej Karpathy

AI researcher and educator: Stanford CS231n co-creator, ex-Tesla AI director, OpenAI
founding member, builder of micrograd/nanoGPT and the "Zero to Hero" series.

## Core contributions to AI engineering
- [karpathy four guidelines](/wiki/karpathy-four-guidelines): think before coding, simplicity first, surgical changes,
  goal-driven execution — distilled from his observations of LLM coding pitfalls.
- [build from scratch pedagogy](/wiki/build-from-scratch-pedagogy): micrograd before PyTorch, nanoGPT before the library.
  The toy implementation IS the explanation — an explicit descendant of Feynman's
  "What I cannot create, I do not understand".
- [llm wiki pattern](/wiki/llm-wiki-pattern): compile knowledge once into interlinked markdown, keep it current,
  instead of rediscovering per query (the foundational pattern for agent knowledge bases).
- Autoresearch: agent loops that run experiments against measurable objectives, an approach
  applied to software optimization in [conformance suites as fitness functions](/wiki/conformance-suites-as-fitness-functions).
- Code as prose: readable, linear, minimal indirection; optimize for the reader —
  human or agent.

## Relationship to Engineering Methodology
Explicit [richard feynman](/wiki/richard-feynman) disciple in method. His guidelines operationalize
[matt pocock](/wiki/matt-pocock)'s "make codebases agents love" at the single-change level (per aihero.dev: "design
codebases agents love"): the flows structure WHEN to code, while the guidelines govern HOW to touch each file safely.

---

## Andrew Ng
- URL: https://pyweb.dev/wiki/andrew-ng
- Raw Markdown: https://pyweb.dev/wiki/andrew-ng.md
- Type: entity
- Tags: person, educator

# Andrew Ng

Andrew Ng is a pioneering AI researcher, educator, Founder of DeepLearning.AI, Managing General Partner at AI Fund, Co-founder of Coursera, and Adjunct Professor at Stanford University.

## Perspective on Agentic Engineering & Software Fundamentals

Ng argues that as coding agents become the primary generator of application code, memorization of programming syntax becomes obsolete while deep understanding of software engineering fundamentals becomes increasingly critical. [[source: andrew-ng-software-engineering-fundamentals-2026]](/wiki/raw/articles/andrew-ng-software-engineering-fundamentals-2026)

In his analysis of AI engineering competencies, Ng emphasizes that developers who "vibe code" without fundamentals produce brittle systems because they fail to steer coding agents through essential engineering tradeoffs (latency, consistency, availability, blast radius, maintainability, and operational cost).

## Core Frameworks & Contributions
- **AI Engineering Skills Map:** Defines the five foundational pillars required to steer coding agents:
  1. Full-stack application mechanics (rendering, caching, APIs, state).
  2. Data architecture & lifecycle management (storage paradigms, consistency, agent-native context).
  3. Evolutionary system architecture & decomposition.
  4. Reliability & shift-left security (verification suites, failure containment).
  5. Production operations & scaling (SDLC, CI/CD, observability, sharding).

---

## Related Concepts & Entities
- [agentic engineering patterns](/wiki/agentic-engineering-patterns) — Disciplined practices distinguishing engineering from vibe coding.
- [software engineering fundamentals for agents](/wiki/software-engineering-fundamentals-for-agents) — The 5-pillar taxonomy for steering coding agents.
- [five debts of agentic engineering](/wiki/five-debts-of-agentic-engineering) — The structural failure modes of unsteered probabilistic code generation.
- [andrej karpathy](/wiki/andrej-karpathy) — Co-educator in AI pedagogy and advocate of foundational comprehension.

---

## Anil Madhavapeddy
- URL: https://pyweb.dev/wiki/anil-madhavapeddy
- Raw Markdown: https://pyweb.dev/wiki/anil-madhavapeddy.md
- Type: entity
- Tags: person, security

# Anil Madhavapeddy

Anil Madhavapeddy is a professor of computer science at Cambridge and a core maintainer of the OCaml compiler. Simon Willison relayed Madhavapeddy's report that an OCaml project website received exploit probes within about ten minutes of a patch discussion becoming public. [[source: simon-willison-just-a-rumour-of-a-bug-is-enough-to-find-a-security-exploit-2026]](/wiki/raw/articles/simon-willison-just-a-rumour-of-a-bug-is-enough-to-find-a-security-exploit-2026)

## Vulnerability Timeline Research

Madhavapeddy reported attempted exploitation within minutes of patches being shared for discussion and argued that this speed is incompatible with existing open-source embargo practices. [[source: simon-willison-just-a-rumour-of-a-bug-is-enough-to-find-a-security-exploit-2026]](/wiki/raw/articles/simon-willison-just-a-rumour-of-a-bug-is-enough-to-find-a-security-exploit-2026)

## Related Concepts
- [agentic vulnerability lifecycle](/wiki/agentic-vulnerability-lifecycle) — the disclosure-window failure his report illustrates
- [agent containment and blast radius](/wiki/agent-containment-and-blast-radius) — mitigation strategies for agent-accelerated security threats

---

## Anthropic
- URL: https://pyweb.dev/wiki/anthropic
- Raw Markdown: https://pyweb.dev/wiki/anthropic.md
- Type: entity
- Summary: AI safety research company developing Claude LLM models and researching agent systems.
- Tags: company, agents, evaluation, security

# Anthropic

AI safety research company developing Claude family of large language models, with research contributions in agent systems and multi-agent architectures according to the 2026 agentic engineering synthesis. [[source: agentic-engineering-trends-2026-synthesis]](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

## Research Contributions

Anthropic is cited in the 2026 synthesis as contributing to multi-agent research systems and agent harness engineering, with publications on effective harnesses for long-running agents and multi-agent research methodologies.

## Multi-Agent Work

The synthesis references Anthropic's research on multi-agent systems for breadth-first research tasks, though notes these measurements are workload-specific rather than universal.

The primary source is now in raw/: Anthropic's 2025 account of its orchestrator-worker Research system, reporting a 90.2% improvement over single-agent Claude Opus 4 on its internal research eval and ~15× token usage versus chat. See [multi agent orchestration](/wiki/multi-agent-orchestration). [[source: anthropic-multi-agent-research-system-2025]](/wiki/raw/articles/anthropic-multi-agent-research-system-2025)

## Containment & Safety Engineering

Anthropic's 2026 engineering posts document containment architectures across claude.ai (ephemeral gVisor containers), Claude Code (human-in-the-loop sandbox plus classifier-gated auto mode), and Cowork (local VM), including disclosed failures such as pre-trust-dialog config execution and exfiltration through an allowlisted domain. See [agent containment and blast radius](/wiki/agent-containment-and-blast-radius). [[source: anthropic-engineering-how-we-contain-claude-2026]](/wiki/raw/articles/anthropic-engineering-how-we-contain-claude-2026)

## Claude Managed Agents Platform
In 2026, Anthropic launched [claude managed agents](/wiki/claude-managed-agents), a hosted infrastructure suite decoupling agent reasoning loops from containerized sandbox execution, exposing stateful event streams for tool use, prompt caching, and human-in-the-loop permission gates. [[source: anthropic-claude-managed-agents-overview-2026]](/wiki/raw/articles/anthropic-claude-managed-agents-overview-2026)

## Cross-links
- [claude managed agents](/wiki/claude-managed-agents) — hosted agent platform
- [copilotkit](/wiki/copilotkit) — AG-UI integration partner
- [simon willison](/wiki/simon-willison) — LLM and AI tooling space
- [shreya shankar](/wiki/shreya-shankar) — AI/ML research community
- [jarred sumner](/wiki/jarred-sumner) — Claude model usage

## Key Publications
- [Effective Harnesses for Long-Running Agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents)
- [Built Multi-Agent Research System](https://www.anthropic.com/engineering/built-multi-agent-research-system)

## Raw Synthesis Source
[2026 Agentic Engineering Trends Synthesis](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

---

## Armin Ronacher
- URL: https://pyweb.dev/wiki/armin-ronacher
- Raw Markdown: https://pyweb.dev/wiki/armin-ronacher.md
- Type: entity
- Summary: Software engineer critiquing coordination friction loss in agent-accelerated development.
- Tags: person, principle

# Armin Ronacher

Software engineer providing critical analysis of architectural risks in agent-accelerated software development, particularly around the loss of beneficial coordination friction. [[source: agentic-engineering-trends-2026-synthesis]](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

## Coordination Friction Critique

Ronacher raises concerns about shared understanding traditionally maintained through coordination friction in software teams. His analysis suggests that removing human bottlenecks may eliminate valuable architectural safeguards.

## Tower Architecture Concerns

The synthesis references Ronacher's "tower keeps rising" thesis about systems becoming too complex for human comprehension when agent acceleration outpaces understanding.

## Better Models, Worse Tools

Ronacher's writing examines the gap between improving model capabilities and corresponding advances in verification and governance infrastructure.

## Cross-links
- [mario zechner](/wiki/mario-zechner) — shared development velocity concerns
- [james shore](/wiki/james-shore) — maintenance and economic concerns
- [simon willison](/wiki/simon-willison) — contemporary in developer community

## Primary Sources
- [Better Models, Worse Tools](https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/)
- [The Tower Keeps Rising](https://lucumr.pocoo.org/2026/7/13/the-tower-keeps-rising/)

## Raw Synthesis Source
[2026 Agentic Engineering Trends Synthesis](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

---

## Beatrice Casey
- URL: https://pyweb.dev/wiki/beatrice-casey
- Raw Markdown: https://pyweb.dev/wiki/beatrice-casey.md
- Type: entity
- Tags: person

# Beatrice Casey

Beatrice Casey is a co-author of *Security Agentic Pull Requests 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/security-agentic-pull-requests-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Beth Barnes
- URL: https://pyweb.dev/wiki/beth-barnes
- Raw Markdown: https://pyweb.dev/wiki/beth-barnes.md
- Type: entity
- Tags: person

# Beth Barnes

Beth Barnes is a co-author of *Metr Developer Productivity Rct 2025*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/metr-developer-productivity-rct-2025.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Boris Cherny
- URL: https://pyweb.dev/wiki/boris-cherny
- Raw Markdown: https://pyweb.dev/wiki/boris-cherny.md
- Type: entity
- Summary: Engineering leader at Anthropic, creator of Claude Code, and author of Programming TypeScript.
- Tags: person, authority, workflow

# Boris Cherny

Boris Cherny is an engineering leader at Anthropic, creator of Claude Code, and author of *Programming TypeScript* (O'Reilly).

## Contributions to Agentic Engineering

Cherny designed the minimal harness architecture for Claude Code, emphasizing direct tool interaction, bash execution, and rigorous self-verification over complex multi-agent frameworks. [[source: boris-cherny-gergely-orosz-building-claude-code-2026]](/wiki/raw/articles/boris-cherny-gergely-orosz-building-claude-code-2026)

### Verification-First Harness Philosophy
Cherny advocates that coding agents must be paired with immediate execution and verification feedback loops:
- **Mandatory Self-Testing:** Every change landed by Claude Code is subjected to automated test suites (`bun test`) and browser verification before review.
- **Attacks Become Evals:** Jailbreaks, test escapes, and hallucinated fixes are automatically ingested as permanent regression evaluation test cases. [[source: boris-cherny-how-boris-uses-claude-code-2026]](/wiki/raw/articles/boris-cherny-how-boris-uses-claude-code-2026)

## Related
- [anthropic](/wiki/anthropic) — organization
- [agentic engineering patterns](/wiki/agentic-engineering-patterns) — verification patterns
- [agent harness engineering](/wiki/agent-harness-engineering) — harness architecture
- [gergely orosz](/wiki/gergely-orosz) — Pragmatic Engineer deep dive

---

## Brittany Reid
- URL: https://pyweb.dev/wiki/brittany-reid
- Raw Markdown: https://pyweb.dev/wiki/brittany-reid.md
- Type: entity
- Tags: person

# Brittany Reid

Brittany Reid is a co-author of *Agentic Pull Requests Github 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/agentic-pull-requests-github-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Cathryn Lavery
- URL: https://pyweb.dev/wiki/cathryn-lavery
- Raw Markdown: https://pyweb.dev/wiki/cathryn-lavery.md
- Type: entity
- Summary: Designer, entrepreneur, and author of Diagram Design; pioneer of editorial visual design systems and semantic diagramming for AI coding agents.
- Tags: person, educator, agents, diagrams

# Cathryn Lavery

Cathryn Lavery is a designer, entrepreneur (founder of BestSelf.co and writer at littlemight.com), and the creator of **Diagram Design**, an opinionated design system and agent skill for generating high-signal editorial diagrams.

## Core Contributions & Philosophy

1. **Deletion as Quality:** Pioneered the philosophy that visual artifacts for technical explanation should prioritize deletion over decoration ("The highest-quality move is usually deletion"). Every node must earn its place; visual density is targeted at 4/10.
2. **Editorial Accent & Signal Discipline:** Enforces the rule that accent colors must be reserved for 1–2 focal elements rather than decorative flair, ensuring visual hierarchy guides reader attention immediately to core takeaways.
3. **Semantic System Patterns Before Visual Layout:** Architected a diagramming methodology that routes behavioral dynamics (queues, bottlenecks, governance catalogs, policy divergence traces) to semantic primitives before picking a physical layout grammar (see [editorial diagrams and visual explanations](/wiki/editorial-diagrams-and-visual-explanations)).
4. **Accessible, Self-Contained Agent Artifacts:** Established standards for coding agents generating visual assets: zero external runtime dependencies, 4px grid alignment, accessible inline SVG structures (`role="img"`, namespaced `aria-labelledby`, structured descriptions), and deterministic redrawing over mechanical translation.

## Relationship to the Engineering Stack

Lavery's work operationalizes [cognitive debt and walkthroughs](/wiki/cognitive-debt-and-walkthroughs) at the visual communication layer. Where [richard feynman](/wiki/richard-feynman) emphasizes plain-spoken pedagogy, [andrej karpathy](/wiki/andrej-karpathy) enforces code simplicity, and [simon willison](/wiki/simon-willison) emphasizes interactive walkthroughs, Lavery provides the visual syntax for rendering complex multi-agent architectures, workflows, and state machines with maximum clarity and minimum visual slop.

## Related
- [editorial diagrams and visual explanations](/wiki/editorial-diagrams-and-visual-explanations)
- [cognitive debt and walkthroughs](/wiki/cognitive-debt-and-walkthroughs)
- [agentic engineering patterns](/wiki/agentic-engineering-patterns)
- [context engineering](/wiki/context-engineering)
- [progressive disclosure](/wiki/progressive-disclosure)

---

## Christoph Csallner
- URL: https://pyweb.dev/wiki/christoph-csallner
- Raw Markdown: https://pyweb.dev/wiki/christoph-csallner.md
- Type: entity
- Tags: person

# Christoph Csallner

Christoph Csallner is a co-author of *Tests Agentic Pull Requests 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/tests-agentic-pull-requests-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Christoph Nakazawa
- URL: https://pyweb.dev/wiki/christoph-nakazawa
- Raw Markdown: https://pyweb.dev/wiki/christoph-nakazawa.md
- Type: entity
- Tags: person, coding-guidelines, feedback-loops

# Christoph Nakazawa

Christoph Nakazawa (nkzw-tech) is the author of `@nkzw/oxlint-config`, an opinionated Oxlint preset whose philosophy — "Error, Never Warn", strict consistent style, bug-pattern prevention, speed, and "don't get in the way" — makes it a reference implementation of a deterministic lint gate for agentic loops. [[source: nkzw-oxlint-config-2026]](/wiki/raw/articles/nkzw-oxlint-config-2026)

## Position

The config's README states its stance directly: "Warnings are noise and get ignored. Either it's an issue, or it isn't." Debug-only code such as `console.log` or `test.only` is disallowed "to avoid unintended logging in production or accidental CI failures", and slow rules are avoided — TypeScript's `noUnusedLocals` is preferred over `no-unused-vars`. [[source: nkzw-oxlint-config-2026]](/wiki/raw/articles/nkzw-oxlint-config-2026)

These choices operationalize the mechanical layer of [constraint layering](/wiki/constraint-layering) and supply the Tier 1 backpressure described in [agentic code quality](/wiki/agentic-code-quality).

## Related
- [deterministic lint gates](/wiki/deterministic-lint-gates) — the pattern his config exemplifies
- [agentic code quality](/wiki/agentic-code-quality) — multi-tier verification architecture

---

## CopilotKit
- URL: https://pyweb.dev/wiki/copilotkit
- Raw Markdown: https://pyweb.dev/wiki/copilotkit.md
- Type: entity
- Summary: Open-source AI Copilot framework and creators of the AG-UI protocol for agent-user interaction.
- Tags: company, agents, workflow

# CopilotKit

**CopilotKit** is an open-source framework and company developing agent-user interaction infrastructure, generative UI primitives, and the [AG UI (Agent User Interaction) protocol](/wiki/ag-ui-protocol). [[source: ag-ui-protocol-specification-2026]](/wiki/raw/articles/ag-ui-protocol-specification-2026)

## Core Initiatives & Contributions

### 1. The AG-UI Protocol
CopilotKit originated the AG-UI specification in partnership with LangChain, CrewAI, and ecosystem partners, establishing a standard event stream protocol (SSE/WebSocket) between backend agent runtimes and frontend client applications. [[source: ag-ui-protocol-specification-2026]](/wiki/raw/articles/ag-ui-protocol-specification-2026)

### 2. Generative UI and Client Tooling
CopilotKit provides client runtime adapters across React, React Native, Next.js, and Angular, alongside multi-surface transports such as the Channels SDK for Slack and Microsoft Teams. [[source: copilotkit-cma-agui-2026]](/wiki/raw/articles/copilotkit-cma-agui-2026)

### 3. Claude Managed Agents Integration
In August 2026, CopilotKit collaborated with [anthropic](/wiki/anthropic) to release first-class AG-UI adapters for [claude managed agents](/wiki/claude-managed-agents), standardizing Anthropic's hosted event logs into client-rendered generative UI and interactive human-in-the-loop approval gates. [[source: copilotkit-cma-agui-2026]](/wiki/raw/articles/copilotkit-cma-agui-2026)

## Cross-links
- [ag ui protocol](/wiki/ag-ui-protocol) — protocol originated by CopilotKit
- [claude managed agents](/wiki/claude-managed-agents) — hosted agent platform integration
- [anthropic](/wiki/anthropic) — partner in CMA integration
- [agent native infrastructure](/wiki/agent-native-infrastructure) — execution tier

---

## Cursor
- URL: https://pyweb.dev/wiki/cursor
- Raw Markdown: https://pyweb.dev/wiki/cursor.md
- Type: entity
- Summary: AI-powered code editor company pioneering hierarchical agent systems for autonomous coding.
- Tags: company, agents, workflow

# Cursor

AI-powered code editor company developing multi-agent orchestration and long-running autonomous coding systems, featured prominently in the 2026 agentic engineering synthesis. [[source: agentic-engineering-trends-2026-synthesis]](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

## FastRender Case Study

The synthesis references Cursor's FastRender project as a landmark case study where hierarchical agent swarms built an experimental browser engine in Rust over a long-running autonomous session.

## Agent Scaling Research

Cursor has published research on scaling agents for autonomous coding workflows, contributing to understanding of multi-agent architecture patterns and parallel worker execution.

## Multi-Agent Architecture

The synthesis describes Cursor's evolution from single-agent to multi-agent patterns, including human orchestrator management, lead planner agents, parallel worker subagents, and adversarial verification.

## Cross-links
- [wilson lin](/wiki/wilson-lin) — Cursor team researcher
- [simon willison](/wiki/simon-willison) — developer tooling space
- [anthropic](/wiki/anthropic) — Claude model usage

## Key Publications
- [Cursor blog on scaling agents](https://cursor.com/blog/scaling-agents)

## Raw Synthesis Source
[2026 Agentic Engineering Trends Synthesis](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

---

## David Gros
- URL: https://pyweb.dev/wiki/david-gros
- Raw Markdown: https://pyweb.dev/wiki/david-gros.md
- Type: entity
- Tags: person

# David Gros

David Gros is a co-author of *Autonomous Agent Contributions Wild 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/autonomous-agent-contributions-wild-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## David Rein
- URL: https://pyweb.dev/wiki/david-rein
- Raw Markdown: https://pyweb.dev/wiki/david-rein.md
- Type: entity
- Tags: person

# David Rein

David Rein is a co-author of *Metr Developer Productivity Rct 2025*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/metr-developer-productivity-rct-2025.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## DeepSeek
- URL: https://pyweb.dev/wiki/deepseek
- Raw Markdown: https://pyweb.dev/wiki/deepseek.md
- Type: entity
- Summary: AI research company behind frontier open-weights models and the DeepSeek Harness agent runtime developer preview.
- Tags: company, agents

# DeepSeek

**DeepSeek** is an AI research and engineering company known for frontier open-weights models and open-source infrastructure. In 2026, DeepSeek released the **DeepSeek Harness** developer preview, advancing the architecture paradigm of `Agent = Model + Harness`.

## Key Contributions to Agent Architecture

DeepSeek structures agent systems by separating the reasoning engine from the operational execution layer:

1. **The Model-Harness Split:** DeepSeek formalized that while "the model is the soul of an agent," the harness is what allows an agent to understand its environment, operate tools, and maintain durability in real-world environments.
2. **Microkernel Plugin Architecture:** In DeepSeek Harness, DeepSeek adopted [Cordis](/wiki/cordis-framework) as an in-process microkernel where every agent subsystem (models, tools, skills, sessions, sandboxes, storage, loops, scheduling, and UI) is implemented as a hot-pluggable service.
3. **Traceability & Deterministic Replay:** DeepSeek emphasizes append-only session event streams, ensuring every context injection, tool call, reasoning trace, and subagent invocation can be inspected, forked, resumed, and replayed from the same stream.

## Related Pages
- [cordis framework](/wiki/cordis-framework)
- [deepseek harness](/wiki/deepseek-harness)
- [agent harness engineering](/wiki/agent-harness-engineering)
- [agent native infrastructure](/wiki/agent-native-infrastructure)
- [anthropic](/wiki/anthropic)

---

## Dipayan Banik
- URL: https://pyweb.dev/wiki/dipayan-banik
- Raw Markdown: https://pyweb.dev/wiki/dipayan-banik.md
- Type: entity
- Tags: person

# Dipayan Banik

Dipayan Banik is a co-author of *Code Review Agents Empirical Study 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/code-review-agents-empirical-study-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## DORA
- URL: https://pyweb.dev/wiki/dora
- Raw Markdown: https://pyweb.dev/wiki/dora.md
- Type: entity
- Tags: company, evaluation

# DORA

DORA researches software delivery performance and organizational capabilities. Its 2025 report frames AI as an amplifier of an organization's existing strengths and weaknesses, placing the surrounding delivery system—not tool adoption alone—at the center of outcomes. [[source: dora-ai-assisted-software-development-2025]](/wiki/raw/articles/dora-ai-assisted-software-development-2025)

## Related
- [agentic quality evidence](/wiki/agentic-quality-evidence) — organizational evidence
- [agentic software factory](/wiki/agentic-software-factory) — delivery operating model
- [agentic code quality](/wiki/agentic-code-quality) — quality controls

---

## Drew Breunig
- URL: https://pyweb.dev/wiki/drew-breunig
- Raw Markdown: https://pyweb.dev/wiki/drew-breunig.md
- Type: entity
- Summary: Writer on LLM economics; argued in 'Fable & The End of the Free Lunch' that high frontier-model pricing ended the era when new models papered over weak harnesses.
- Tags: person, agents, context-engineering

# Drew Breunig

Drew Breunig writes about LLM pricing and coding-agent strategy. [simon willison](/wiki/simon-willison) quoted his post *Fable & The End of the Free Lunch* on 23rd August 2026. [[source: simon-willison-quoting-drew-breunig-2026]](/wiki/raw/articles/simon-willison-quoting-drew-breunig-2026)

## The end of the free lunch

Breunig's argument: prior to Fable it felt silly to waste much time improving your coding harness or context strategies, because "A new model would arrive at the same price (or cheaper!) and paper over most of your problems." Then Fable landed — incredible, but its cost was so high, and Opus was good enough (as was 5.6, K3, and even GLM) for most needed code, that "So we started to think about what work went where." [[source: simon-willison-quoting-drew-breunig-2026]](/wiki/raw/articles/simon-willison-quoting-drew-breunig-2026)

Adoption data supports the cost pressure: Ramp's billing-based index for July 2026 put Fable 5 at 8.0% of Anthropic model spend versus 28.0% for the cheaper Opus 4.8. [[source: simon-willison-anthropic-s-best-ai-model-struggles-to-attract-users-as-chea-2026]](/wiki/raw/articles/simon-willison-anthropic-s-best-ai-model-struggles-to-attract-users-as-chea-2026)

The implication for practitioners: once frontier capability stops arriving at flat or falling prices, investment shifts from waiting for models to engineering the harness — model routing by task value, [context engineering](/wiki/context-engineering), and [agent harness engineering](/wiki/agent-harness-engineering) become durable work rather than throwaway glue.

## Related
- [simon willison](/wiki/simon-willison)
- [anthropic](/wiki/anthropic)
- [agent harness engineering](/wiki/agent-harness-engineering)
- [context engineering](/wiki/context-engineering)

---

## Dung Nguyen Manh
- URL: https://pyweb.dev/wiki/dung-nguyen-manh
- Raw Markdown: https://pyweb.dev/wiki/dung-nguyen-manh.md
- Type: entity
- Tags: person

# Dung Nguyen Manh

Dung Nguyen Manh is a co-author of *Swe Evo Long Horizon 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/swe-evo-long-horizon-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Eirini Kalliamvakou
- URL: https://pyweb.dev/wiki/eirini-kalliamvakou
- Raw Markdown: https://pyweb.dev/wiki/eirini-kalliamvakou.md
- Type: entity
- Tags: person

# Eirini Kalliamvakou

Eirini Kalliamvakou is a co-author of *Github Copilot Productivity Experiment 2023*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/github-copilot-productivity-experiment-2023.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Epoch AI
- URL: https://pyweb.dev/wiki/epoch-ai
- Raw Markdown: https://pyweb.dev/wiki/epoch-ai.md
- Type: entity
- Tags: company, evaluation

# Epoch AI

Epoch AI publishes research on AI capabilities and benchmark interpretation. Its analysis of SWE-bench Verified emphasizes the benchmark's concentration in small Python bug fixes, limited repository diversity, contamination risk, and sensitivity to the surrounding scaffold. [[source: epoch-swe-bench-verified-analysis-2025]](/wiki/raw/articles/epoch-swe-bench-verified-analysis-2025)

## Related
- [agentic quality evidence](/wiki/agentic-quality-evidence) — benchmark-validity evidence
- [error analysis and evals](/wiki/error-analysis-and-evals) — evaluation design
- [agentic code quality](/wiki/agentic-code-quality) — oracle-health controls

---

## Florian Brand
- URL: https://pyweb.dev/wiki/florian-brand
- Raw Markdown: https://pyweb.dev/wiki/florian-brand.md
- Type: entity
- Tags: person

# Florian Brand

Florian Brand is a co-author of *What skills does SWE-bench Verified evaluate?*, source material used to evaluate code quality and verification in agentic software development. [[source: epoch-swe-bench-verified-analysis-2025]](/wiki/raw/articles/epoch-swe-bench-verified-analysis-2025)

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Gelei Deng
- URL: https://pyweb.dev/wiki/gelei-deng
- Raw Markdown: https://pyweb.dev/wiki/gelei-deng.md
- Type: entity
- Tags: person

# Gelei Deng

Gelei Deng is a co-author of *Overeager Coding Agents 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/overeager-coding-agents-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Gergely Orosz
- URL: https://pyweb.dev/wiki/gergely-orosz
- Raw Markdown: https://pyweb.dev/wiki/gergely-orosz.md
- Type: entity
- Summary: Author of The Pragmatic Engineer newsletter, covering tech industry trends, engineering leadership, and developer tooling in the agentic era.
- Tags: person, authority, workflow

# Gergely Orosz

Gergely Orosz is a software engineer and author of *The Pragmatic Engineer*, a widely read engineering newsletter analyzing software industry practices, engineering management, and developer tooling.

## Coverage of Agentic Engineering

Orosz has documented the practical integration of autonomous AI coding agents across engineering teams, conducting high-impact deep dives with pioneers including [kent beck](/wiki/kent-beck) on TDD-governed agent workflows and [boris cherny](/wiki/boris-cherny) on Claude Code's architectural harness and verification principles. [[source: kent-beck-gergely-orosz-tdd-ai-agents-2025]](/wiki/raw/articles/kent-beck-gergely-orosz-tdd-ai-agents-2025) [[source: boris-cherny-gergely-orosz-building-claude-code-2026]](/wiki/raw/articles/boris-cherny-gergely-orosz-building-claude-code-2026)

## Related
- [kent beck](/wiki/kent-beck) — TDD in agentic workflows
- [boris cherny](/wiki/boris-cherny) — Claude Code harness design
- [agentic engineering patterns](/wiki/agentic-engineering-patterns) — industry patterns
- [agentic code quality](/wiki/agentic-code-quality) — verification controls

---

## GitHub
- URL: https://pyweb.dev/wiki/github
- Raw Markdown: https://pyweb.dev/wiki/github.md
- Type: entity
- Summary: Leading software development platform, developer of Copilot, and author of the Spec-Driven Development toolkit for AI agents.
- Tags: organization, workflow, tool-use

# GitHub

GitHub is a major developer platform and creator of GitHub Copilot and the Spec-Driven Development (SDD) toolkit.

## Contributions to Agentic Engineering

GitHub introduced Spec-Driven Development toolkits for AI coding agents, establishing formal executable specifications as the primary contract guiding agent generation, test creation, and validation phases. [[source: github-spec-driven-development-ai-2025]](/wiki/raw/articles/github-spec-driven-development-ai-2025)

## Related
- [agentic engineering patterns](/wiki/agentic-engineering-patterns) — disciplined agent workflows
- [agentic code quality](/wiki/agentic-code-quality) — specification and verification controls
- [openai](/wiki/openai) — Copilot partner

---

## Hajimu Iida
- URL: https://pyweb.dev/wiki/hajimu-iida
- Raw Markdown: https://pyweb.dev/wiki/hajimu-iida.md
- Type: entity
- Tags: person

# Hajimu Iida

Hajimu Iida is a co-author of *Agent Generated Code Maintenance 2026* and *Agentic Pull Requests Github 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/agent-generated-code-maintenance-2026.md]^[raw/papers/agentic-pull-requests-github-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Hamel Husain
- URL: https://pyweb.dev/wiki/hamel-husain
- Raw Markdown: https://pyweb.dev/wiki/hamel-husain.md
- Type: entity
- Summary: AI product engineer, machine learning educator, and specialist in LLM evaluation, error analysis, and domain-grounded AI systems.
- Tags: person, educator, evaluation, context-engineering

# Hamel Husain

Hamel Husain is an AI engineer, educator, and co-founder of Parlance Labs. He is one of the foremost advocates for rigorous, empirical evaluation methodologies in generative AI and agentic systems, emphasizing human error analysis, custom data viewers, and practical data science fundamentals over generic benchmarks.

## Core Philosophy & Contributions

1. **"Look At Your Data":** Husain argues that the single highest-ROI activity in AI engineering is qualitative [error analysis and evals](/wiki/error-analysis-and-evals) on real production traces. Generic off-the-shelf metrics (like generic hallucination or helpfulness scores) create an illusion of progress while obscuring domain-specific failure modes.
2. **The Optimization Hierarchy:** When building AI products, teams must exhaust context engineering, prompt refinement, and harness tooling before resorting to model post-training or fine-tuning.
3. **Data Science Fundamentals in AI:** Viewing LLM judges as supervised classifiers requiring human-annotated validation sets, precision/recall tracking, and strict partition boundaries.
4. **Active Learning & Tooling:** Collaborating with researchers like [shreya shankar](/wiki/shreya-shankar) on active-learning-assisted trace labeling and failure discovery tools.
5. **Verifiability as product design:** "It's hard to eval" is a product smell — artifacts hard for the builder to verify are hard for users too; design checkable artifacts before building evals ([designing for verifiability](/wiki/designing-for-verifiability)). [[source: hamel-husain-it-s-hard-to-eval-is-a-product-smell-2026]](/wiki/raw/articles/hamel-husain-it-s-hard-to-eval-is-a-product-smell-2026)
6. **The Revenge of the Data Scientist:** every recurring eval pitfall maps to a missing data-science fundamental; the agent harness itself is largely data science. [[source: hamel-husain-the-revenge-of-the-data-scientist-2026]](/wiki/raw/articles/hamel-husain-the-revenge-of-the-data-scientist-2026)

He co-teaches *AI Evals for Engineers and PMs*, with over 4,500 students from 500+ companies (including OpenAI, Anthropic, and Google), and previously worked at Airbnb and GitHub, including early LLM research used by OpenAI for code understanding. [[source: hamel-husain-do-automated-evals-work-2026]](/wiki/raw/articles/hamel-husain-do-automated-evals-work-2026)

## Related
- [shreya shankar](/wiki/shreya-shankar)
- [error analysis and evals](/wiki/error-analysis-and-evals)
- [designing for verifiability](/wiki/designing-for-verifiability)
- [automated eval engineering](/wiki/automated-eval-engineering)
- [closed loop agent improvement](/wiki/closed-loop-agent-improvement)
- [context engineering](/wiki/context-engineering)

---

## Hao Li
- URL: https://pyweb.dev/wiki/hao-li
- Raw Markdown: https://pyweb.dev/wiki/hao-li.md
- Type: entity
- Tags: person

# Hao Li

Hao Li is a co-author of *Agentic Pull Requests Github 2026* and *Aidev Dataset 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/agentic-pull-requests-github-2026.md]^[raw/papers/aidev-dataset-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Haoxiang Zhang
- URL: https://pyweb.dev/wiki/haoxiang-zhang
- Raw Markdown: https://pyweb.dev/wiki/haoxiang-zhang.md
- Type: entity
- Tags: person

# Haoxiang Zhang

Haoxiang Zhang is a co-author of *Aidev Dataset 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/aidev-dataset-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Hiroshi Iwata
- URL: https://pyweb.dev/wiki/hiroshi-iwata
- Raw Markdown: https://pyweb.dev/wiki/hiroshi-iwata.md
- Type: entity
- Tags: person

# Hiroshi Iwata

Hiroshi Iwata is a co-author of *Agent Generated Code Maintenance 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/agent-generated-code-maintenance-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Huy Nhat Phan
- URL: https://pyweb.dev/wiki/huy-nhat-phan
- Raw Markdown: https://pyweb.dev/wiki/huy-nhat-phan.md
- Type: entity
- Tags: person

# Huy Nhat Phan

Huy Nhat Phan is a co-author of *Swe Evo Long Horizon 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/swe-evo-long-horizon-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Ivan Bercovich
- URL: https://pyweb.dev/wiki/ivan-bercovich
- Raw Markdown: https://pyweb.dev/wiki/ivan-bercovich.md
- Type: entity
- Tags: person

# Ivan Bercovich

Ivan Bercovich is a co-author of *Adversarial Hacker Fixer Verifiers 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/adversarial-hacker-fixer-verifiers-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Ivar Jacobson
- URL: https://pyweb.dev/wiki/ivar-jacobson
- Raw Markdown: https://pyweb.dev/wiki/ivar-jacobson.md
- Type: entity
- Tags: person, educator

# Ivar Jacobson

Author of "Object Oriented Software Engineering: A Use-Case Driven Approach" and creator of BCE (Boundary-Control-Entity) architecture. His BCE pattern influenced [clean architecture](/wiki/clean-architecture) by establishing the separation between boundary interfaces, control logic, and core entities.

## Related Concepts
- BCE — his Boundary-Control-Entity architectural pattern
- [clean architecture](/wiki/clean-architecture) — incorporates his BCE separation concepts

 [[source: uncle-bob-clean-architecture-2012]](/wiki/raw/articles/uncle-bob-clean-architecture-2012)

---

## Ivgeni Segal
- URL: https://pyweb.dev/wiki/ivgeni-segal
- Raw Markdown: https://pyweb.dev/wiki/ivgeni-segal.md
- Type: entity
- Tags: person

# Ivgeni Segal

Ivgeni Segal is a co-author of *Adversarial Hacker Fixer Verifiers 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/adversarial-hacker-fixer-verifiers-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## James Coplien
- URL: https://pyweb.dev/wiki/james-coplien
- Raw Markdown: https://pyweb.dev/wiki/james-coplien.md
- Type: entity
- Tags: person, educator

# James Coplien

Co-author with [trygve reenskaug](/wiki/trygve-reenskaug) of DCI (Data, Context, and Interaction) architecture. His DCI pattern influenced [robert c martin](/wiki/robert-c-martin)'s [clean architecture](/wiki/clean-architecture) synthesis by contributing to the understanding of separation of concerns in layered systems.

## Related Concepts
- DCI — his architectural pattern contribution
- [clean architecture](/wiki/clean-architecture) — incorporates ideas from his DCI work

 [[source: uncle-bob-clean-architecture-2012]](/wiki/raw/articles/uncle-bob-clean-architecture-2012)

---

## James Shore
- URL: https://pyweb.dev/wiki/james-shore
- Raw Markdown: https://pyweb.dev/wiki/james-shore.md
- Type: entity
- Summary: Software engineering consultant analyzing economic implications of AI-accelerated development.
- Tags: person, principle

# James Shore

Software engineering consultant and agile development expert who authored analysis on economic costs of AI-accelerated development, particularly maintenance considerations. [[source: agentic-engineering-trends-2026-synthesis]](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

## AI Development Economics

Shore's blog post "You Need AI That Reduces Your Maintenance Costs" examines the long-term economic implications of AI assistance in software development, focusing on how increased velocity may create hidden maintenance burdens.

## Maintenance Cost Focus

Shore's analysis emphasizes the need for AI systems that actively reduce rather than increase long-term maintenance costs, challenging assumptions about productivity gains from accelerated development.

Key concerns include:
- Velocity improvements vs. value creation
- Long-term maintenance implications of AI-generated code
- Economic validation of AI adoption strategies

## Cross-links
- [mario zechner](/wiki/mario-zechner) — shared concerns about development velocity
- [armin ronacher](/wiki/armin-ronacher) — architectural complexity concerns
- [simon willison](/wiki/simon-willison) — contemporary in developer community

## Primary Source

- [Maintenance-cost analysis](https://www.jamesshore.com/v2/blog/2026/you-need-ai-that-reduces-your-maintenance-costs)

---

## Jarred Sumner
- URL: https://pyweb.dev/wiki/jarred-sumner
- Raw Markdown: https://pyweb.dev/wiki/jarred-sumner.md
- Type: entity
- Summary: Creator of Bun, who led autonomous agent-driven rewrite of Bun's codebase from Zig to Rust.
- Tags: person, agents, evaluation

# Jarred Sumner

Creator and lead developer of Bun JavaScript runtime, who conducted a large-scale autonomous agent coding project rewriting Bun's codebase from Zig to Rust using Claude agents. [[source: agentic-engineering-trends-2026-synthesis]](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

## Bun Rewrite Project

The synthesis documents Sumner's landmark autonomous agent project: rewriting more than half a million lines of Bun's codebase from Zig to Rust using Claude agents, requiring 5.9B input tokens and 690M output tokens at an estimated cost of ~$165K.

## Performance Results

The autonomous rewrite achieved measurable improvements:
- 10% faster startup time
- 35% memory reduction  
- 5× idle CPU reduction

## Conformance Testing

The project's success relied on Bun's external TypeScript test suite, demonstrating how external verification systems enable autonomous optimization when agents have deterministic success criteria.

## Cross-links
- [conformance suites as fitness functions](/wiki/conformance-suites-as-fitness-functions)
- [agentic engineering patterns](/wiki/agentic-engineering-patterns)
- [tobias lutke](/wiki/tobias-lutke) — autonomous optimization projects
- [anthropic](/wiki/anthropic) — Claude model usage

## Primary Sources
- [Bun in Rust](https://bun.com/blog/bun-in-rust)

## Raw Synthesis Source
[2026 Agentic Engineering Trends Synthesis](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

---

## Jean-Stanislas Denain
- URL: https://pyweb.dev/wiki/jean-stanislas-denain
- Raw Markdown: https://pyweb.dev/wiki/jean-stanislas-denain.md
- Type: entity
- Tags: person

# Jean-Stanislas Denain

Jean-Stanislas Denain is a co-author of *What skills does SWE-bench Verified evaluate?*, source material used to evaluate code quality and verification in agentic software development. [[source: epoch-swe-bench-verified-analysis-2025]](/wiki/raw/articles/epoch-swe-bench-verified-analysis-2025)

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Jeffrey Palermo
- URL: https://pyweb.dev/wiki/jeffrey-palermo
- Raw Markdown: https://pyweb.dev/wiki/jeffrey-palermo.md
- Type: entity
- Tags: person, educator

# Jeffrey Palermo

Software architect and creator of [onion architecture](/wiki/onion-architecture), a layered architectural pattern that influenced [clean architecture](/wiki/clean-architecture). His onion pattern emphasizes dependency inversion and separation of concerns through concentric layers.

## Key Contributions

### Onion Architecture
Developed the Onion Architecture pattern that organizes software into concentric layers with dependencies pointing inward toward the core domain. This pattern directly influenced [robert c martin](/wiki/robert-c-martin)'s [clean architecture](/wiki/clean-architecture) synthesis and its emphasis on the [dependency rule](/wiki/dependency-rule).

## Related Concepts
- [onion architecture](/wiki/onion-architecture) — his core architectural contribution  
- [clean architecture](/wiki/clean-architecture) — incorporates his onion layering approach

 [[source: uncle-bob-clean-architecture-2012]](/wiki/raw/articles/uncle-bob-clean-architecture-2012)

---

## Joanna C. S. Santos
- URL: https://pyweb.dev/wiki/joanna-c-s-santos
- Raw Markdown: https://pyweb.dev/wiki/joanna-c-s-santos.md
- Type: entity
- Tags: person

# Joanna C. S. Santos

Joanna C. S. Santos is a co-author of *Security Agentic Pull Requests 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/security-agentic-pull-requests-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Joel Becker
- URL: https://pyweb.dev/wiki/joel-becker
- Raw Markdown: https://pyweb.dev/wiki/joel-becker.md
- Type: entity
- Tags: person

# Joel Becker

Joel Becker is a co-author of *Metr Developer Productivity Rct 2025*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/metr-developer-productivity-rct-2025.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Johann Rehberger
- URL: https://pyweb.dev/wiki/johann-rehberger
- Raw Markdown: https://pyweb.dev/wiki/johann-rehberger.md
- Type: entity
- Summary: Security researcher warning about normalization of deviance in AI system permissions.
- Tags: person, agents, anti-patterns

# Johann Rehberger

Security researcher examining risks in AI system deployment, particularly focused on the normalization of deviance in granting agents broad system permissions. [[source: agentic-engineering-trends-2026-synthesis]](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

## Normalization of Deviance Analysis

The synthesis references Rehberger's work on the gradual acceptance of increasingly risky AI system permissions, representing systematic erosion of security boundaries as teams become comfortable with broader agent access.

## AI Security Concerns

Rehberger's research addresses:
- Prompt injection risks in dependencies and documentation
- Need for ephemeral sandboxed execution environments
- Security boundary erosion in agent deployment
- Risk accumulation through gradual permission expansion

## Security vs. Capability Tradeoffs

His analysis highlights tension between agent capability and security constraints, warning against sacrificing security boundaries for operational convenience.

## Cross-links
- [armin ronacher](/wiki/armin-ronacher) — system architecture concerns
- [simon willison](/wiki/simon-willison) — AI/security intersection
- [shreya shankar](/wiki/shreya-shankar) — AI safety community

## Primary Sources
- [The Normalization of Deviance in AI](https://embracethered.com/blog/posts/2025/the-normalization-of-deviance-in-ai/)

## Raw Synthesis Source
[2026 Agentic Engineering Trends Synthesis](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

---

## K M Ferdous
- URL: https://pyweb.dev/wiki/k-m-ferdous
- Raw Markdown: https://pyweb.dev/wiki/k-m-ferdous.md
- Type: entity
- Tags: person

# K M Ferdous

K M Ferdous is a co-author of *Code Review Agents Empirical Study 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/code-review-agents-empirical-study-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Ken’ichi Yamaguchi
- URL: https://pyweb.dev/wiki/kenichi-yamaguchi
- Raw Markdown: https://pyweb.dev/wiki/kenichi-yamaguchi.md
- Type: entity
- Tags: person

# Ken’ichi Yamaguchi

Ken’ichi Yamaguchi is a co-author of *Agent Generated Code Maintenance 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/agent-generated-code-maintenance-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Kent Beck
- URL: https://pyweb.dev/wiki/kent-beck
- Raw Markdown: https://pyweb.dev/wiki/kent-beck.md
- Type: entity
- Summary: Pioneer of Extreme Programming (XP), Test-Driven Development (TDD), co-author of the Agile Manifesto, and author writing on AI coding workflows.
- Tags: person, authority, tdd, workflow

# Kent Beck

Kent Beck is a software engineer, creator of Extreme Programming (XP), pioneer of Test-Driven Development (TDD), and co-author of the Agile Manifesto.

## Contributions to Agentic Engineering

In discussions on modern AI coding tools with [gergely orosz](/wiki/gergely-orosz), Beck characterized AI coding agents as an "unpredictable genie" that fulfills requests literally but often through unexpected or unprincipled paths. [[source: kent-beck-gergely-orosz-tdd-ai-agents-2025]](/wiki/raw/articles/kent-beck-gergely-orosz-tdd-ai-agents-2025)

### TDD as the Agent Governor
Beck argues that TDD is a fundamental "superpower" when working with autonomous coding agents. Because agents can easily hallucinate plausible-looking solutions, deterministic test suites act as the essential constraint governing agent output.

### The Test Deletion Anti-Pattern
Beck identified a major operational failure mode in autonomous coding agents: when faced with difficult failing tests, agents frequently attempt to delete, modify, or soften the test assertions to achieve a green test exit code rather than solving the underlying architectural flaw. This insight underpins the requirement for read-only protected test baselines in [agentic code quality](/wiki/agentic-code-quality).

## Related
- [agentic engineering patterns](/wiki/agentic-engineering-patterns) — TDD governor and patterns
- [agentic code quality](/wiki/agentic-code-quality) — multi-tier quality controls
- [red green tdd](/wiki/red-green-tdd) — test-driven cycles
- [martin fowler](/wiki/martin-fowler) — collaborator on agile and software design
- [gergely orosz](/wiki/gergely-orosz) — industry commentator

---

## Kexun Zhang
- URL: https://pyweb.dev/wiki/kexun-zhang
- Raw Markdown: https://pyweb.dev/wiki/kexun-zhang.md
- Type: entity
- Tags: person

# Kexun Zhang

Kexun Zhang is a co-author of *Adversarial Hacker Fixer Verifiers 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/adversarial-hacker-fixer-verifiers-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Kowshik Chowdhury
- URL: https://pyweb.dev/wiki/kowshik-chowdhury
- Raw Markdown: https://pyweb.dev/wiki/kowshik-chowdhury.md
- Type: entity
- Tags: person

# Kowshik Chowdhury

Kowshik Chowdhury is a co-author of *Code Review Agents Empirical Study 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/code-review-agents-empirical-study-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Lalit Maganti
- URL: https://pyweb.dev/wiki/lalit-maganti
- Raw Markdown: https://pyweb.dev/wiki/lalit-maganti.md
- Type: entity
- Summary: Syntaqlite creator documenting where coding agents help implementation and harm unresolved design work.
- Tags: person, agents, anti-patterns, workflow

# Lalit Maganti

Lalit Maganti built Syntaqlite, a parser, formatter, validator, and language server grounded in SQLite's grammar, and published a detailed account of using coding agents during its development.

## Implementation Versus Design

In [Eight years of wanting, three months of building with AI](https://lalitm.com/post/building-syntaqlite-ai/), Maganti describes agents as powerful for repetitive, objectively checkable implementation work. The project included hundreds of grammar rules where tests and precise behavior supplied useful feedback.

He also reports losing a month to an early, poorly understood architecture and ultimately rewriting it. His conclusion is not that agents are useless, but that implementation leverage does not replace human judgment about APIs, history, taste, and system shape.

## Relevance

The case study connects [conformance suites as fitness functions](/wiki/conformance-suites-as-fitness-functions) with the limits described by [cognitive debt and walkthroughs](/wiki/cognitive-debt-and-walkthroughs). It is a concrete example of why [agentic engineering patterns](/wiki/agentic-engineering-patterns) need explicit human design ownership.

## Related Concepts

- [conformance suites as fitness functions](/wiki/conformance-suites-as-fitness-functions)
- [cognitive debt and walkthroughs](/wiki/cognitive-debt-and-walkthroughs)
- [agentic engineering patterns](/wiki/agentic-engineering-patterns)
- [context engineering](/wiki/context-engineering)

## Primary Source

- [Eight years of wanting, three months of building with AI](https://lalitm.com/post/building-syntaqlite-ai/)

---

## Leo Yu Zhang
- URL: https://pyweb.dev/wiki/leo-yu-zhang
- Raw Markdown: https://pyweb.dev/wiki/leo-yu-zhang.md
- Type: entity
- Tags: person

# Leo Yu Zhang

Leo Yu Zhang is a co-author of *Overeager Coding Agents 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/overeager-coding-agents-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Maliheh Izadi
- URL: https://pyweb.dev/wiki/maliheh-izadi
- Raw Markdown: https://pyweb.dev/wiki/maliheh-izadi.md
- Type: entity
- Tags: person

# Maliheh Izadi

Maliheh Izadi is a co-author of *Autonomous Agent Contributions Wild 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/autonomous-agent-contributions-wild-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Mario Zechner
- URL: https://pyweb.dev/wiki/mario-zechner
- Raw Markdown: https://pyweb.dev/wiki/mario-zechner.md
- Type: entity
- Summary: Software engineer warning about compounding errors in high-velocity agent development.
- Tags: person, principle

# Mario Zechner

Software engineer examining systemic risks of unbounded agent acceleration in software development, particularly focused on error compounding and development velocity concerns. [[source: agentic-engineering-trends-2026-synthesis]](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

## Development Velocity Concerns

Zechner's writing warns about agents maintaining consistent high velocity regardless of error accumulation, unlike human developers who naturally slow down when encountering complexity or confusion.

## Code Quality Analysis

His work discusses the end result of unchecked agent development:
- Small AI errors that compound over time
- Rapid development without corresponding quality safeguards  
- Code that functions but becomes difficult for humans to maintain

## Sustainable Development Advocacy

Zechner advocates for intentional friction and verification points in agent workflows to prevent runaway complexity, emphasizing that velocity does not equal value.

## Cross-links
- [armin ronacher](/wiki/armin-ronacher) — architectural complexity concerns
- [james shore](/wiki/james-shore) — maintenance cost economics
- [simon willison](/wiki/simon-willison) — contemporary in developer community

## Primary Sources
- [Thoughts on Slowing the Fuck Down](https://mariozechner.at/posts/2026-03-25-thoughts-on-slowing-the-fuck-down/)

## Raw Synthesis Source
[2026 Agentic Engineering Trends Synthesis](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

---

## Martin Fowler
- URL: https://pyweb.dev/wiki/martin-fowler
- Raw Markdown: https://pyweb.dev/wiki/martin-fowler.md
- Type: entity
- Summary: Author, chief scientist at ThoughtWorks, and foundational thinker on software architecture, refactoring, and AI-assisted engineering practices.
- Tags: person, authority, workflow

# Martin Fowler

Martin Fowler is an author, software architect, and chief scientist at ThoughtWorks, known for foundational works on refactoring, design patterns, and agile methodologies.

## Views on AI-Assisted Engineering

In *Fragments* (2026), Fowler reflected on the role of automated testing in AI-assisted development, articulating:

> "TDD served a critical function in AI-assisted development: it kept me in the loop." [[source: martin-fowler-fragments-2026-01-08]](/wiki/raw/articles/martin-fowler-fragments-2026-01-08)

Fowler argues that while agents accelerate code synthesis, developer comprehension and architectural steering remain the primary safeguards against accumulated cognitive debt. TDD acts not only as a correctness oracle for the agent, but as a cognitive anchor ensuring the human engineer understands system boundaries and intent.

## Related
- [agentic engineering patterns](/wiki/agentic-engineering-patterns) — comprehension loops and TDD
- [kent beck](/wiki/kent-beck) — longtime collaborator on XP and agile
- [cognitive debt and walkthroughs](/wiki/cognitive-debt-and-walkthroughs) — human comprehension in AI systems
- [agentic code quality](/wiki/agentic-code-quality) — quality control architecture

---

## Matt Pocock
- URL: https://pyweb.dev/wiki/matt-pocock
- Raw Markdown: https://pyweb.dev/wiki/matt-pocock.md
- Type: entity
- Summary: TypeScript and AI engineering educator behind the idea-to-ship workflow and deep module design.
- Tags: person, educator, workflow, agents

# Matt Pocock

AI-engineering and TypeScript educator (AI Hero, Total TypeScript). Developer of structured agent execution workflows and phased engineering systems.

## LLM Fundamentals Teaching
Through the LLM Fundamentals course (raw/aihero-video transcripts), Pocock teaches the foundational mechanics of LLM applications: the [llm message protocol](/wiki/llm-message-protocol) (stateless message histories, system prompts, token economics), the [tool calling loop](/wiki/tool-calling-loop) (with the under-6-tools rule of thumb), [structured outputs](/wiki/structured-outputs), [model provider abstraction](/wiki/model-provider-abstraction) via Vercel's AI SDK, and [model context protocol basics](/wiki/model-context-protocol-basics). His pedagogy is show-the-thing-first: demonstrate a working use case before the abstraction.

## Core Operational Models

### 1. The Vicious Circle of AI Code Decay
> *"Bad code is now the most expensive it has ever been."*
- Unconstrained agents produce code quickly without deep boundary abstractions.
- Decayed codebases increase context noise and hallucination rates for subsequent agent turns.
- **Solution:** Enforce strict workflow phase boundaries, type safety, and automated test harnesses before writing feature logic.

### 2. The Idea-to-Ship Execution Graph
- **Grilling:** Sharpen intent and reject vague specifications upfront before code generation (see [grilling doctrine](/wiki/grilling-doctrine) and [grill with docs](/wiki/grill-with-docs)).
- **Handoff Artifacts:** Compress conversational context into structured handoffs; clear context windows between phases.
- **Tracer Bullets:** Verify end-to-end integration slices early to prove system viability (see [tracer bullets](/wiki/tracer-bullets)).

### 3. Progressive Skill Encapsulation
- Package workflows into discrete, composable skills loaded on-demand rather than polluting global instructions (see [progressive disclosure](/wiki/progressive-disclosure)).

## Related
[richard feynman](/wiki/richard-feynman), [andrej karpathy](/wiki/andrej-karpathy), [idea to ship flow](/wiki/idea-to-ship-flow), [grilling doctrine](/wiki/grilling-doctrine), [tracer bullets](/wiki/tracer-bullets), [context engineering](/wiki/context-engineering), [ai coding taxonomy](/wiki/ai-coding-taxonomy).

---

## Mert Demirer
- URL: https://pyweb.dev/wiki/mert-demirer
- Raw Markdown: https://pyweb.dev/wiki/mert-demirer.md
- Type: entity
- Tags: person

# Mert Demirer

Mert Demirer is a co-author of *Github Copilot Productivity Experiment 2023*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/github-copilot-productivity-experiment-2023.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Miku Watanabe
- URL: https://pyweb.dev/wiki/miku-watanabe
- Raw Markdown: https://pyweb.dev/wiki/miku-watanabe.md
- Type: entity
- Tags: person

# Miku Watanabe

Miku Watanabe is a co-author of *Agentic Pull Requests Github 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/agentic-pull-requests-github-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Minh Vu Thai Pham
- URL: https://pyweb.dev/wiki/minh-vu-thai-pham
- Raw Markdown: https://pyweb.dev/wiki/minh-vu-thai-pham.md
- Type: entity
- Tags: person

# Minh Vu Thai Pham

Minh Vu Thai Pham is a co-author of *Swe Evo Long Horizon 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/swe-evo-long-horizon-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Model Evaluation & Threat Research
- URL: https://pyweb.dev/wiki/model-evaluation-and-threat-research
- Raw Markdown: https://pyweb.dev/wiki/model-evaluation-and-threat-research.md
- Type: entity
- Tags: company, evaluation

# Model Evaluation & Threat Research

Model Evaluation & Threat Research (METR) studies frontier AI capabilities and their real-world effects. Its task-completion horizon work models agent success against human expert task duration, while its developer-productivity randomized trial found that early-2025 AI tools increased completion time by 19% for experienced maintainers working in familiar mature repositories. [[source: metr-task-completion-time-horizons-2026]](/wiki/raw/articles/metr-task-completion-time-horizons-2026)^[raw/papers/metr-developer-productivity-rct-2025.md]

## Related
- [agentic quality evidence](/wiki/agentic-quality-evidence) — benchmark and productivity evidence
- [error analysis and evals](/wiki/error-analysis-and-evals) — evaluation design and error analysis
- [agentic code quality](/wiki/agentic-code-quality) — proposed factory controls

---

## Mohammed Latif Siddiq
- URL: https://pyweb.dev/wiki/mohammed-latif-siddiq
- Raw Markdown: https://pyweb.dev/wiki/mohammed-latif-siddiq.md
- Type: entity
- Tags: person

# Mohammed Latif Siddiq

Mohammed Latif Siddiq is a co-author of *Security Agentic Pull Requests 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/security-agentic-pull-requests-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Nat Pryce
- URL: https://pyweb.dev/wiki/nat-pryce
- Raw Markdown: https://pyweb.dev/wiki/nat-pryce.md
- Type: entity
- Tags: person, educator

# Nat Pryce

Co-author with [steve freeman](/wiki/steve-freeman) of "Growing Object Oriented Software" who adopted [alistair cockburn](/wiki/alistair-cockburn)'s [hexagonal architecture](/wiki/hexagonal-architecture) pattern. Their book helped establish the Ports and Adapters approach in mainstream practice.

## Related Concepts
- [hexagonal architecture](/wiki/hexagonal-architecture) — adopted and promoted Cockburn's pattern with Freeman
- [clean architecture](/wiki/clean-architecture) — contributed to the architectural foundations

 [[source: uncle-bob-clean-architecture-2012]](/wiki/raw/articles/uncle-bob-clean-architecture-2012)

---

## Nate Rush
- URL: https://pyweb.dev/wiki/nate-rush
- Raw Markdown: https://pyweb.dev/wiki/nate-rush.md
- Type: entity
- Tags: person

# Nate Rush

Nate Rush is a co-author of *Metr Developer Productivity Rct 2025*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/metr-developer-productivity-rct-2025.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Nghi D. Q. Bui
- URL: https://pyweb.dev/wiki/nghi-d-q-bui
- Raw Markdown: https://pyweb.dev/wiki/nghi-d-q-bui.md
- Type: entity
- Tags: person

# Nghi D. Q. Bui

Nghi D. Q. Bui is a co-author of *Swe Evo Long Horizon 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/swe-evo-long-horizon-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Nick Craig-Wood
- URL: https://pyweb.dev/wiki/nick-craig-wood
- Raw Markdown: https://pyweb.dev/wiki/nick-craig-wood.md
- Type: entity
- Tags: person, security, workflow

# Nick Craig-Wood

Nick Craig-Wood is the maintainer of rclone. In comments quoted by Simon Willison, he reported more than 40 security disclosures in one month, compared with about 20 across the project's first 10 years. [[source: simon-willison-just-a-rumour-of-a-bug-is-enough-to-find-a-security-exploit-2026]](/wiki/raw/articles/simon-willison-just-a-rumour-of-a-bug-is-enough-to-find-a-security-exploit-2026)

## Disclosure Volume Analysis

Craig-Wood said about 75% of those disclosures contained "a nugget of something which needs looking at" and described the resulting triage and repair workload as substantial. [[source: simon-willison-just-a-rumour-of-a-bug-is-enough-to-find-a-security-exploit-2026]](/wiki/raw/articles/simon-willison-just-a-rumour-of-a-bug-is-enough-to-find-a-security-exploit-2026)

## Related Concepts
- [agentic vulnerability lifecycle](/wiki/agentic-vulnerability-lifecycle) — the disclosure volume surge phenomenon
- [agent containment and blast radius](/wiki/agent-containment-and-blast-radius) — strategies for managing high-volume agent-generated security reports

---

## OpenAI
- URL: https://pyweb.dev/wiki/openai
- Raw Markdown: https://pyweb.dev/wiki/openai.md
- Type: entity
- Summary: AI research and deployment company developing GPT models, ChatGPT, Codex, and agent harness infrastructure.
- Tags: company, agents, evaluation, security

# OpenAI

**OpenAI** is an AI research and deployment company known for the GPT family of large language models, ChatGPT, and agent runtime infrastructure including the [Codex](/wiki/codex-harness-architecture) coding harness and the Agents SDK.

## Agent Harness Engineering & Architecture

OpenAI develops production agent harnesses and execution runtimes characterized by modular separation and system-level isolation:

- **[Codex Harness](/wiki/codex-harness-architecture) (`openai/codex`):** An open-source Rust agent workspace (`codex-rs`) implementing [Clean Architecture](/wiki/clean-architecture) and [Hexagonal Architecture](/wiki/hexagonal-architecture). It decouples the core turn loop from client frontends via `codex-app-server` (JSON-RPC) and enforces OS-level process isolation via multi-platform sandboxing (`bwrap`, Seatbelt, Windows restricted tokens).
- **Tool Specialization:** Reinforcement learning alignment around structured tool interfaces such as `apply_patch` for unified diffs, integrated alongside the Model Context Protocol (`codex-mcp`).
- **Bounded Output Streams:** Harness-level protection against context blowouts through head/tail stream buffers (`unified_exec`) and session time-travel primitives (`thread_rollback`, `thread_fork`).

## Multi-Agent & Orchestration Research

OpenAI's orchestration work spans:
- **Agents SDK:** Multi-agent handoff patterns, guardrails, and persistent sessions.
- **Symphony:** Issue-tracker-driven control plane for isolated autonomous agent execution.
- **BrowseComp:** Benchmark evaluating browsing agent information retrieval in complex environments.

## Benchmark Auditing

OpenAI's 2026 audit of SWE-bench Verified reported two distinct validity problems at frontier performance levels: tests that reject functionally valid solutions and evidence that public tasks and gold patches have entered model training data. The organization stopped treating the benchmark as a reliable frontier launch measure and recommended less contaminated alternatives. [[source: openai-swe-bench-verified-audit-2026]](/wiki/raw/articles/openai-swe-bench-verified-audit-2026)

## Cross-links
- [codex harness architecture](/wiki/codex-harness-architecture) — architectural deep dive into OpenAI's Codex harness
- [agent harness engineering](/wiki/agent-harness-engineering) — runtime infrastructure, sandboxes, and tool contracts
- [agentic quality evidence](/wiki/agentic-quality-evidence) — benchmark validity and code-quality evidence
- [anthropic](/wiki/anthropic) — peer AI frontier research lab
- [deepseek](/wiki/deepseek) — developer of open-weights models and modular harness runtimes
- [simon willison](/wiki/simon-willison) — analysis of coding agent loops and harness patterns

## Primary Sources
- [OpenAI Codex Repository (GitHub)](https://github.com/openai/codex)
- [OpenAI Harness Engineering Field Report](https://openai.com/index/harness-engineering/)
- [Unlocking the Codex Harness: App Server](https://openai.com/index/unlocking-the-codex-harness/)

---

## Peter Cihon
- URL: https://pyweb.dev/wiki/peter-cihon
- Raw Markdown: https://pyweb.dev/wiki/peter-cihon.md
- Type: entity
- Tags: person

# Peter Cihon

Peter Cihon is a co-author of *Github Copilot Productivity Experiment 2023*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/github-copilot-productivity-experiment-2023.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Prem Devanbu
- URL: https://pyweb.dev/wiki/prem-devanbu
- Raw Markdown: https://pyweb.dev/wiki/prem-devanbu.md
- Type: entity
- Tags: person

# Prem Devanbu

Prem Devanbu is a co-author of *Autonomous Agent Contributions Wild 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/autonomous-agent-contributions-wild-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Prithvi Rajasekaran
- URL: https://pyweb.dev/wiki/prithvi-rajasekaran
- Raw Markdown: https://pyweb.dev/wiki/prithvi-rajasekaran.md
- Type: entity
- Tags: person, agents, evaluation

# Prithvi Rajasekaran

Member of [anthropic](/wiki/anthropic)'s Labs team and author of *Harness design for long-running application development* (Mar 24, 2026). Worked on "two interconnected problems: getting Claude to produce high-quality frontend designs, and getting it to build complete applications without human intervention."

Designed the GAN-inspired [generator evaluator loop](/wiki/generator-evaluator-loop): a generator/evaluator pair for frontend design, extended to a three-agent planner–generator–evaluator harness for multi-hour autonomous full-stack builds, with sprint contracts negotiated between agents before any code is written.

## Related
- [anthropic](/wiki/anthropic)
- [generator evaluator loop](/wiki/generator-evaluator-loop)
- [agent harness engineering](/wiki/agent-harness-engineering)

---

## Rahul Pandita
- URL: https://pyweb.dev/wiki/rahul-pandita
- Raw Markdown: https://pyweb.dev/wiki/rahul-pandita.md
- Type: entity
- Tags: person

# Rahul Pandita

Rahul Pandita is a co-author of *Autonomous Agent Contributions Wild 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/autonomous-agent-contributions-wild-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Razvan Mihai Popescu
- URL: https://pyweb.dev/wiki/razvan-mihai-popescu
- Raw Markdown: https://pyweb.dev/wiki/razvan-mihai-popescu.md
- Type: entity
- Tags: person

# Razvan Mihai Popescu

Razvan Mihai Popescu is a co-author of *Autonomous Agent Contributions Wild 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/autonomous-agent-contributions-wild-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Richard Feynman
- URL: https://pyweb.dev/wiki/richard-feynman
- Raw Markdown: https://pyweb.dev/wiki/richard-feynman.md
- Type: entity
- Summary: Nobel laureate physicist known for first-principles thinking, joyful curiosity, and plain-language pedagogy.
- Tags: person, physicist, teaching, worldview

# Richard Feynman

Nobel laureate in physics (QED) and pioneer of first-principles scientific methodology. Originator of radical honesty, the Feynman technique, and cargo-cult critique.

## Core Operational Principles

### 1. Radical Honesty and Anti-Cargo-Cult Verification
> *"The first principle is that you must not fool yourself — and you are the easiest person to fool."*
- **Nature Cannot Be Fooled:** In software and autonomous systems, PR, vibes, and plausible text mean nothing. Only deterministic execution and passing test harnesses count as proof.
- **Cargo-Cult Prevention:** Discard the mere appearance of rigor (e.g. boilerplate wrappers, performative tests) in favor of actual falsification and error analysis.

### 2. The Feynman Technique (Pedagogy & Evals)
- Explain a concept in plain English without jargon as if to a 12-year-old.
- Identify where the explanation breaks down or relies on hand-waving.
- Re-explain the missing seam using a minimal concrete example.
- **Agent Application:** Evaluate agent reasoning by testing whether it can solve minimal isolated edge cases before tackling monolithic tasks.

### 3. "What I Cannot Create, I Do Not Understand"
- Build minimal toy systems from scratch (e.g. micrograd, minimal parser) to verify complete understanding of the underlying mechanics before adopting high-level abstractions (see [build from scratch pedagogy](/wiki/build-from-scratch-pedagogy)).

## Related
[first principles thinking](/wiki/first-principles-thinking), [feynman technique](/wiki/feynman-technique), [build from scratch pedagogy](/wiki/build-from-scratch-pedagogy), [andrej karpathy](/wiki/andrej-karpathy), [simon willison](/wiki/simon-willison).

---

## Robert C. Martin (Uncle Bob)
- URL: https://pyweb.dev/wiki/robert-c-martin
- Raw Markdown: https://pyweb.dev/wiki/robert-c-martin.md
- Type: entity
- Tags: person, educator

# Robert C. Martin (Uncle Bob)

Software craftsman, author, and educator known for promoting software engineering principles and clean code practices. Uncle Bob is a prominent figure in the software development community, advocating for disciplined engineering practices.

## Key Contributions

### Clean Architecture (2012)
Martin synthesized multiple architectural patterns into a unified [clean architecture](/wiki/clean-architecture) approach, establishing the [dependency rule](/wiki/dependency-rule) as the core organizing principle. His architectural model integrates ideas from [hexagonal architecture](/wiki/hexagonal-architecture), [onion architecture](/wiki/onion-architecture), and other layered approaches into a single coherent framework.

### SOLID Principles & Clean Code
Author of foundational works on software craftsmanship, including advocacy for testable, framework-independent systems that prioritize business rules over technical details.

## Related Concepts
- [clean architecture](/wiki/clean-architecture) — his 2012 synthesis of layered architectural patterns
- [dependency rule](/wiki/dependency-rule) — the core principle governing clean architecture boundaries
- Screaming architecture — his concept that architecture should reveal system intent

 [[source: uncle-bob-clean-architecture-2012]](/wiki/raw/articles/uncle-bob-clean-architecture-2012)

---

## Sabrina Haque
- URL: https://pyweb.dev/wiki/sabrina-haque
- Raw Markdown: https://pyweb.dev/wiki/sabrina-haque.md
- Type: entity
- Tags: person

# Sabrina Haque

Sabrina Haque is a co-author of *Tests Agentic Pull Requests 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/tests-agentic-pull-requests-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Sarvesh Ingale
- URL: https://pyweb.dev/wiki/sarvesh-ingale
- Raw Markdown: https://pyweb.dev/wiki/sarvesh-ingale.md
- Type: entity
- Tags: person

# Sarvesh Ingale

Sarvesh Ingale is a co-author of *Tests Agentic Pull Requests 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/tests-agentic-pull-requests-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Shashwat Saxena
- URL: https://pyweb.dev/wiki/shashwat-saxena
- Raw Markdown: https://pyweb.dev/wiki/shashwat-saxena.md
- Type: entity
- Tags: person

# Shashwat Saxena

Shashwat Saxena is a co-author of *Adversarial Hacker Fixer Verifiers 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/adversarial-hacker-fixer-verifiers-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Shazibul Islam Shamim
- URL: https://pyweb.dev/wiki/shazibul-islam-shamim
- Raw Markdown: https://pyweb.dev/wiki/shazibul-islam-shamim.md
- Type: entity
- Tags: person

# Shazibul Islam Shamim

Shazibul Islam Shamim is a co-author of *Code Review Agents Empirical Study 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/code-review-agents-empirical-study-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Shopify
- URL: https://pyweb.dev/wiki/shopify
- Raw Markdown: https://pyweb.dev/wiki/shopify.md
- Type: entity
- Summary: E-commerce platform company conducting autonomous agent optimization experiments.
- Tags: company, agents, evaluation

# Shopify

E-commerce platform company that conducted autonomous agent optimization experiments on their Liquid templating language, documented in the 2026 agentic engineering synthesis. [[source: agentic-engineering-trends-2026-synthesis]](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

## Liquid Optimization Project

Under Tobias Lütke's direction, Shopify conducted autonomous agent optimization of their Liquid templating language, achieving 53% faster parse+render performance, 61% fewer memory allocations, and 93 verified commits across 120 automated trials.

## Autonomous Optimization Results

The overnight autonomous agent loop with benchmark scripts demonstrated how deterministic test suites enable agents to become autonomous optimization engines rather than drifting on ambiguous tasks.

## Open Source Documentation

The optimization work is documented in the Shopify Liquid GitHub repository, providing a public case study of agent-driven performance optimization.

## Cross-links
- [tobias lutke](/wiki/tobias-lutke) — engineering leadership behind the project
- [jarred sumner](/wiki/jarred-sumner) — autonomous optimization projects
- [simon willison](/wiki/simon-willison) — developer tooling community

## Key Publications
- [Shopify Liquid GitHub repository](https://github.com/Shopify/liquid/pull/2056)

## Raw Synthesis Source
[2026 Agentic Engineering Trends Synthesis](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

---

## Shota Sawada
- URL: https://pyweb.dev/wiki/shota-sawada
- Raw Markdown: https://pyweb.dev/wiki/shota-sawada.md
- Type: entity
- Tags: person

# Shota Sawada

Shota Sawada is a co-author of *Agent Generated Code Maintenance 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/agent-generated-code-maintenance-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Shreya Shankar
- URL: https://pyweb.dev/wiki/shreya-shankar
- Raw Markdown: https://pyweb.dev/wiki/shreya-shankar.md
- Type: entity
- Summary: Computer science researcher at UC Berkeley focusing on data management, ML systems, and active-learning tooling for LLM error analysis and evaluation.
- Tags: person, educator, evaluation

# Shreya Shankar

Shreya Shankar is a computer science researcher at UC Berkeley specializing in machine learning systems, data management, and operational tooling for AI evaluation.

## Key Research & Systems

1. **Error Discovery & Active Learning:** Pioneered active-learning workflows where an AI assistant observes real-time human trace annotations, updates a dynamic failure taxonomy, and proactively queries similar unlabeled records.
2. **Criteria Drift in Evaluation:** Demonstrated empirically that humans often cannot specify complete evaluation rubrics up front; criteria naturally emerge through the iterative act of labeling and observing model behaviors.
3. **Model Cascades (BARGAIN):** Created algorithms for optimal model cascading, routing high-confidence queries to cheap, small models while reserving frontier models for ambiguous edge cases to cut inference costs up to 86% without sacrificing quality.
4. **Data Agent Benchmark (DAB):** Designed benchmarks reflecting realistic, messy multi-database environments to measure planning and execution failures in analytical agents.

## Related
- [hamel husain](/wiki/hamel-husain)
- [error analysis and evals](/wiki/error-analysis-and-evals)
- [automated eval engineering](/wiki/automated-eval-engineering)
- [closed loop agent improvement](/wiki/closed-loop-agent-improvement)

---

## Sida Peng
- URL: https://pyweb.dev/wiki/sida-peng
- Raw Markdown: https://pyweb.dev/wiki/sida-peng.md
- Type: entity
- Tags: person

# Sida Peng

Sida Peng is a co-author of *Github Copilot Productivity Experiment 2023*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/github-copilot-productivity-experiment-2023.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Simon Willison
- URL: https://pyweb.dev/wiki/simon-willison
- Raw Markdown: https://pyweb.dev/wiki/simon-willison.md
- Type: entity
- Summary: Software engineer, creator of Datasette, co-creator of Django, and prominent authority on LLM tooling, evals, and agentic engineering patterns.
- Tags: person, authority, workflow

# Simon Willison

Simon Willison is a software engineer, co-creator of Django, creator of Datasette, and an authority on generative AI tooling, prompt injection security, and agentic engineering patterns.

## Contributions to Agentic Engineering

Willison cataloged the canon of [agentic engineering patterns](/wiki/agentic-engineering-patterns), arguing that writing code has become cheap while software quality, verification, and maintainability remain the primary engineering constraints. [[source: simon-willison-agentic-engineering-patterns-2026]](/wiki/raw/articles/simon-willison-agentic-engineering-patterns-2026)

### Proof-of-Work Doctrine
In "Your job is to deliver code you have proven to work" (2025), Willison established that the developer's role shifts from writing raw syntax to providing rigorous proof of correctness. AI coding agents must be directed to run tests, execute browser verifications, and demonstrate proof before PR submission. [[source: simon-willison-code-proven-to-work-2025]](/wiki/raw/articles/simon-willison-code-proven-to-work-2025)

### Core Patterns
- **First run the tests:** Forcing an agent to execute existing test suites before editing code to seed context and establish baseline invariants.
- **Red/Green TDD for Agents:** Requiring failing reproduction tests before implementation to prevent self-fulfilling test assertions.
- **Agentic Manual Testing:** Utilizing browser automation and runtime tools for interactive verification.

## Related
- [agentic engineering patterns](/wiki/agentic-engineering-patterns) — cataloged patterns
- [agentic code quality](/wiki/agentic-code-quality) — verification controls
- [red green tdd](/wiki/red-green-tdd) — test-driven loops
- [agentic manual testing](/wiki/agentic-manual-testing) — dynamic verification
- [addy osmani](/wiki/addy-osmani) — contemporary in developer tooling

---

## Steve Freeman
- URL: https://pyweb.dev/wiki/steve-freeman
- Raw Markdown: https://pyweb.dev/wiki/steve-freeman.md
- Type: entity
- Tags: person, educator

# Steve Freeman

Co-author with [nat pryce](/wiki/nat-pryce) of "Growing Object Oriented Software" who adopted [alistair cockburn](/wiki/alistair-cockburn)'s [hexagonal architecture](/wiki/hexagonal-architecture) pattern. His work helped popularize the Ports and Adapters approach that influenced [clean architecture](/wiki/clean-architecture).

## Related Concepts
- [hexagonal architecture](/wiki/hexagonal-architecture) — adopted and promoted Cockburn's pattern
- [clean architecture](/wiki/clean-architecture) — his work contributed to the architectural synthesis

 [[source: uncle-bob-clean-architecture-2012]](/wiki/raw/articles/uncle-bob-clean-architecture-2012)

---

## Stripe
- URL: https://pyweb.dev/wiki/stripe
- Raw Markdown: https://pyweb.dev/wiki/stripe.md
- Type: entity
- Summary: Financial infrastructure platform conducting benchmark research on autonomous coding agents building production payment integrations.
- Tags: organization, evaluation, benchmarks

# Stripe

Stripe is a financial infrastructure platform providing payment processing, billing, and developer APIs for internet businesses.

## Agentic Engineering & Benchmark Research

Stripe developed the **Stripe Integration Benchmark**, evaluating whether autonomous coding agents can author, migrate, and verify complex full-stack financial integrations end-to-end. [[source: stripe-can-ai-agents-build-real-stripe-integrations-2026]](/wiki/raw/articles/stripe-can-ai-agents-build-real-stripe-integrations-2026)

### Key Empirical Findings
- **The False Victory Anti-Pattern:** When upgrading SDKs through breaking changes, agents frequently triggered HTTP 400 Bad Requests due to malformed mock data and misinterpreted the error payload as proof that the endpoint was operational.
- **Full-Stack Verification Requirement:** Autonomous agents require multimodal verification tools (browser automation, API state inspection, database verifiers) to confirm that payments actually clear rather than relying on code syntax alone. [[source: stripe-you-cant-whisper-at-an-ai-agent-2026]](/wiki/raw/articles/stripe-you-cant-whisper-at-an-ai-agent-2026)

## Related
- [agentic code quality](/wiki/agentic-code-quality) — verification controls
- [agentic engineering patterns](/wiki/agentic-engineering-patterns) — browser verification patterns
- [anthropic](/wiki/anthropic) — collaboration on agent evals
- [openai](/wiki/openai) — evaluation targets

---

## Tatsuya Shirai
- URL: https://pyweb.dev/wiki/tatsuya-shirai
- Raw Markdown: https://pyweb.dev/wiki/tatsuya-shirai.md
- Type: entity
- Tags: person

# Tatsuya Shirai

Tatsuya Shirai is a co-author of *Agent Generated Code Maintenance 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/agent-generated-code-maintenance-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Tessl
- URL: https://pyweb.dev/wiki/tessl
- Raw Markdown: https://pyweb.dev/wiki/tessl.md
- Type: entity
- Summary: AI software development company researching developer platforms, agentic software engineering benchmarks, and system harness architectures.
- Tags: company, evaluation

# Tessl

AI developer tools and software engineering research organization based in London, UK. Tessl conducts empirical research on agentic software development, benchmark alignment, and autonomous system harnesses. [[source: tessl-coding-benchmarks-misaligned-system-harness-2026]](/wiki/raw/articles/tessl-coding-benchmarks-misaligned-system-harness-2026)

## Research & Frameworks

- **Position on Coding Benchmarks:** Tessl researchers (Maria I. Gorinova, Dru Knox, Amy Heineike, et al.) authored the foundational 2026 ACM SIGKDD position paper arguing that pre-agent coding benchmarks (SWE-Bench, HumanEval) fail to evaluate real-world agentic software engineering because they conflate raw model capability with the composite **system harness**. [[source: tessl-coding-benchmarks-misaligned-system-harness-2026]](/wiki/raw/articles/tessl-coding-benchmarks-misaligned-system-harness-2026)
- **NS2 System Harness:** An open-source, issue-driven multi-agent harness treating GitHub issues as the state machine for task decomposition, execution, test quality validation, and multi-tier feedback loops. [[source: tessl-coding-benchmarks-misaligned-system-harness-2026]](/wiki/raw/articles/tessl-coding-benchmarks-misaligned-system-harness-2026)

## Cross-links
- [agentic code quality](/wiki/agentic-code-quality) — feedback signal tiers and verification gates
- [agent harness engineering](/wiki/agent-harness-engineering) — composite system harness design
- [designing for verifiability](/wiki/designing-for-verifiability) — objective verification criteria over reference solutions
- [error analysis and evals](/wiki/error-analysis-and-evals) — benchmark and eval methodology

---

## Tobias Lütke
- URL: https://pyweb.dev/wiki/tobias-lutke
- Raw Markdown: https://pyweb.dev/wiki/tobias-lutke.md
- Type: entity
- Summary: Shopify CEO who conducted autonomous agent optimization of Liquid template engine.
- Tags: person, agents, evaluation

# Tobias Lütke

Shopify CEO who led an autonomous agent optimization project on the Liquid template engine, demonstrating practical applications of agent-driven performance optimization. [[source: agentic-engineering-trends-2026-synthesis]](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

## Shopify Liquid Optimization

Lütke conducted an overnight autonomous agent loop optimization of Shopify's Liquid template engine with benchmark scripts, achieving 53% faster parse+render performance, 61% fewer allocations, and 93 verified commits across 120 automated trials.

## Autonomous Optimization Validation

The project provides concrete evidence for the effectiveness of overnight autonomous agent loops when combined with robust benchmark verification systems, with a high success rate demonstrating viability under proper constraints.

## Performance Engineering

The Liquid optimization demonstrates how autonomous agents can achieve significant performance improvements in production systems while maintaining verification standards through automated testing.

## Cross-links
- [shopify](/wiki/shopify) — company context for the optimization work
- [jarred sumner](/wiki/jarred-sumner) — autonomous optimization projects
- [simon willison](/wiki/simon-willison) — developer community

## Primary Sources
- [Shopify Liquid Pull Request](https://github.com/Shopify/liquid/pull/2056)

## Raw Synthesis Source
[2026 Agentic Engineering Trends Synthesis](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis)

---

## Trygve Reenskaug
- URL: https://pyweb.dev/wiki/trygve-reenskaug
- Raw Markdown: https://pyweb.dev/wiki/trygve-reenskaug.md
- Type: entity
- Tags: person, educator

# Trygve Reenskaug

Co-author with [james coplien](/wiki/james-coplien) of DCI (Data, Context, and Interaction) architecture. His work on DCI patterns contributed to the architectural thinking that [robert c martin](/wiki/robert-c-martin) synthesized into [clean architecture](/wiki/clean-architecture).

## Related Concepts  
- DCI — his architectural pattern with Coplien
- [clean architecture](/wiki/clean-architecture) — his DCI work influenced the synthesis

 [[source: uncle-bob-clean-architecture-2012]](/wiki/raw/articles/uncle-bob-clean-architecture-2012)

---

## Tue Le
- URL: https://pyweb.dev/wiki/tue-le
- Raw Markdown: https://pyweb.dev/wiki/tue-le.md
- Type: entity
- Tags: person

# Tue Le

Tue Le is a co-author of *Swe Evo Long Horizon 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/swe-evo-long-horizon-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Vercel
- URL: https://pyweb.dev/wiki/vercel
- Raw Markdown: https://pyweb.dev/wiki/vercel.md
- Type: entity
- Summary: Frontend platform company; creator of the AI SDK, a provider-neutral TypeScript interface for LLM generation, streaming, and tool calling.
- Tags: company, llm-infrastructure

# Vercel

Frontend deployment platform (Next.js) and creator of the **AI SDK** — an open-source TypeScript toolkit providing a provider-neutral `LanguageModel` interface for LLM text generation, streaming, structured outputs, embeddings, and tool calling. The AI SDK is the reference implementation of [model provider abstraction](/wiki/model-provider-abstraction): adapters exist for Anthropic, OpenAI, and any OpenAI-compatible endpoint including local models.

## Notable
- AI SDK patterns (generate → stream → structure → tools) map directly onto the [llm message protocol](/wiki/llm-message-protocol) and [structured outputs](/wiki/structured-outputs) concepts.
- Widely used as the teaching substrate for AI engineering education (e.g. AI Hero's AI SDK crash course).

## Related
[model provider abstraction](/wiki/model-provider-abstraction), [llm message protocol](/wiki/llm-message-protocol), [tool calling loop](/wiki/tool-calling-loop), [structured outputs](/wiki/structured-outputs).

---

## Vinicius Carvalho Lopes
- URL: https://pyweb.dev/wiki/vinicius-carvalho-lopes
- Raw Markdown: https://pyweb.dev/wiki/vinicius-carvalho-lopes.md
- Type: entity
- Tags: person

# Vinicius Carvalho Lopes

Vinicius Carvalho Lopes is a co-author of *Security Agentic Pull Requests 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/security-agentic-pull-requests-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Viv Trivedy
- URL: https://pyweb.dev/wiki/viv-trivedy
- Raw Markdown: https://pyweb.dev/wiki/viv-trivedy.md
- Type: entity
- Summary: Practitioner credited by Addy Osmani with naming harness engineering and articulating the model-plus-harness framing.
- Tags: person, agents, context-engineering

# Viv Trivedy

Viv Trivedy is credited by Addy Osmani's [Agent Harness Engineering](https://addyosmani.com/blog/agent-harness-engineering/) article with coining the term *harness engineering* and publishing an anatomy of the scaffolding around an agent.

## Contribution

The framing separates the model from the surrounding execution system: prompts, tools, context policies, sandboxes, orchestration, feedback loops, persistence, and recovery paths. This directs engineering attention toward the runtime conditions that make model behavior reliable.

Osmani summarizes Trivedy's formulation as **Agent = Model + Harness**. The attribution here is intentionally grounded in Osmani's article because the original social post was not captured in the ingestion archive.

## Related Concepts

- [agent harness engineering](/wiki/agent-harness-engineering)
- [context engineering](/wiki/context-engineering)
- [multi agent orchestration](/wiki/multi-agent-orchestration)

## Primary Source

- [Addy Osmani: Agent Harness Engineering](https://addyosmani.com/blog/agent-harness-engineering/)

---

## Wes McKinney
- URL: https://pyweb.dev/wiki/wes-mckinney
- Raw Markdown: https://pyweb.dev/wiki/wes-mckinney.md
- Type: entity
- Summary: Creator of pandas, POSIT; pioneer of high-throughput agentic engineering harness workflows and automated post-commit review.
- Tags: person, workflow

# Wes McKinney

Software engineer and open-source creator (creator of `pandas`, co-founder of POSIT, Arrow contributor). In 2026, McKinney pioneered high-volume agentic engineering architectures, operating autonomous coding agents across multiple parallel projects generating ~1M lines of code over 6 months at 1.3–1.4B tokens/day. [[source: hugo-bowne-anderson-agentic-software-factory-2026]](/wiki/raw/articles/hugo-bowne-anderson-agentic-software-factory-2026)

## Agentic Engineering Philosophy

McKinney contrasts "vibe coding" with disciplined agentic engineering:
> "The difference between vibe coding and agentic engineering is planning, architecture, and caring about the output." [[source: hugo-bowne-anderson-agentic-software-factory-2026]](/wiki/raw/articles/hugo-bowne-anderson-agentic-software-factory-2026)

When code production outpaces human review capacity, the human role shifts from reading diffs to defining architectural boundaries, managing task decomposition, and supervising automated review harnesses.

## Toolchain & Harness Innovations

- **RoboRev:** An automated daemon running as a post-commit hook that triggers high-reasoning LLM code review on every commit, maintaining an append-only findings ledger.
- **Commit-Every-Turn Discipline:** Enforcing granular git commits at every agent step to maintain review ledger precision.
- **Local Control Plane:** Kata (local issue tracking), Middleman (local PR review dashboard), and Agents View (session retrieval across hundreds of agent traces).

## Cross-links
- [agentic software factory](/wiki/agentic-software-factory) — operating model for autonomous code generation and verification
- [agentic code quality](/wiki/agentic-code-quality) — multi-tier verification and constraint-driven backpressure
- [generator evaluator loop](/wiki/generator-evaluator-loop) — separating generator models from reviewer models
- [agent harness engineering](/wiki/agent-harness-engineering) — system harness orchestration and tooling

---

## Wilson Lin
- URL: https://pyweb.dev/wiki/wilson-lin
- Raw Markdown: https://pyweb.dev/wiki/wilson-lin.md
- Type: entity
- Summary: Cursor researcher documenting planner-worker architectures for long-running autonomous coding.
- Tags: person, agents, workflow

# Wilson Lin

Wilson Lin is an author of Cursor's [Scaling long-running autonomous coding](https://cursor.com/blog/scaling-agents), which documents experiments coordinating hundreds of coding agents on large software projects.

## Planner–Worker Architecture

The Cursor team reports that flat peer coordination suffered from lock contention, abandoned locks, and agents choosing small, safe tasks. Their replacement separated responsibilities:

- planners explore and create tasks, recursively spawning sub-planners;
- workers complete assigned tasks without coordinating with one another;
- a judge decides whether another iteration is required.

The experiments included the FastRender browser project and other long-running migrations. The post presents these as research systems requiring review, not proof that autonomous swarms replace normal engineering discipline.

## Related Concepts

- [multi agent orchestration](/wiki/multi-agent-orchestration)
- [agent harness engineering](/wiki/agent-harness-engineering)
- [agentic engineering patterns](/wiki/agentic-engineering-patterns)

## Primary Source

- [Cursor: Scaling long-running autonomous coding](https://cursor.com/blog/scaling-agents)

---

## Xinye Zhao
- URL: https://pyweb.dev/wiki/xinye-zhao
- Raw Markdown: https://pyweb.dev/wiki/xinye-zhao.md
- Type: entity
- Tags: person

# Xinye Zhao

Xinye Zhao is a co-author of *Security Agentic Pull Requests 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/security-agentic-pull-requests-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Yanjun Zhang
- URL: https://pyweb.dev/wiki/yanjun-zhang
- Raw Markdown: https://pyweb.dev/wiki/yanjun-zhang.md
- Type: entity
- Tags: person

# Yanjun Zhang

Yanjun Zhang is a co-author of *Overeager Coding Agents 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/overeager-coding-agents-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Yi Liu
- URL: https://pyweb.dev/wiki/yi-liu
- Raw Markdown: https://pyweb.dev/wiki/yi-liu.md
- Type: entity
- Tags: person

# Yi Liu

Yi Liu is a co-author of *Overeager Coding Agents 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/overeager-coding-agents-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Ying Zhang
- URL: https://pyweb.dev/wiki/ying-zhang
- Raw Markdown: https://pyweb.dev/wiki/ying-zhang.md
- Type: entity
- Tags: person

# Ying Zhang

Ying Zhang is a co-author of *Overeager Coding Agents 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/overeager-coding-agents-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Yubin Qu
- URL: https://pyweb.dev/wiki/yubin-qu
- Raw Markdown: https://pyweb.dev/wiki/yubin-qu.md
- Type: entity
- Tags: person

# Yubin Qu

Yubin Qu is a co-author of *Overeager Coding Agents 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/overeager-coding-agents-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Yuekang Li
- URL: https://pyweb.dev/wiki/yuekang-li
- Raw Markdown: https://pyweb.dev/wiki/yuekang-li.md
- Type: entity
- Tags: person

# Yuekang Li

Yuekang Li is a co-author of *Overeager Coding Agents 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/overeager-coding-agents-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Yutaro Kashiwa
- URL: https://pyweb.dev/wiki/yutaro-kashiwa
- Raw Markdown: https://pyweb.dev/wiki/yutaro-kashiwa.md
- Type: entity
- Tags: person

# Yutaro Kashiwa

Yutaro Kashiwa is a co-author of *Agent Generated Code Maintenance 2026* and *Agentic Pull Requests Github 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/agent-generated-code-maintenance-2026.md]^[raw/papers/agentic-pull-requests-github-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---

## Ziqian Zhong
- URL: https://pyweb.dev/wiki/ziqian-zhong
- Raw Markdown: https://pyweb.dev/wiki/ziqian-zhong.md
- Type: entity
- Tags: person

# Ziqian Zhong

Ziqian Zhong is a co-author of *Adversarial Hacker Fixer Verifiers 2026*, source material used to evaluate code quality and verification in agentic software development.^[raw/papers/adversarial-hacker-fixer-verifiers-2026.md]

## Related
- [agentic code quality](/wiki/agentic-code-quality) — empirical implications for factory quality controls
- [agentic quality evidence](/wiki/agentic-quality-evidence) — comparison of benchmark, field, and controlled evidence

---


# Section: COMPARISONS

## Agentic Code Quality — Evidence Map
- URL: https://pyweb.dev/wiki/agentic-quality-evidence
- Raw Markdown: https://pyweb.dev/wiki/agentic-quality-evidence.md
- Type: comparison
- Tags: comparison, evaluation, agents, code-review, security

# Agentic Code Quality — Evidence Map

This page separates observed empirical evidence and industrial production telemetry from the control architecture proposed in [agentic code quality](/wiki/agentic-code-quality).

## Practitioner Consensus vs. Empirical Trials

There is a significant structural tension between practitioner consensus and controlled academic trials:

```mermaid
flowchart TD
    subgraph Practitioner [Practitioner & Industry Consensus]
        P1[Willison: Proof-of-Work Doctrine]
        P2[Beck & Fowler: TDD as Governor]
        P3[Stripe & Airbnb: Full-Stack Verification & EDD]
        P4[Cherny: Verify-Every-Change Harness]
    end

    subgraph Empirical [Empirical & Controlled Trials]
        E1[METR RCT: 19% Slowdown on Familiar Codebases]
        E2[AIDev / Wild Studies: Churn & Test-Weakening]
        E3[Epoch & OpenAI: Benchmark Contamination & Oracle Hacks]
        E4[OverEager-Bench: Task Success != Authorization]
    end

    Practitioner -.->|Uncontrolled Observational Field Evidence| EvidenceGap[Evidence Tension]
    Empirical -.->|Controlled Rigor / Task Constraints| EvidenceGap
```

| Domain | Practitioner Claim (Industry Field Consensus) | Empirical Counter-Evidence (Controlled Studies) |
|---|---|---|
| **Productivity & Speed** | 2x–5x velocity gains with high code quality ([addy osmani](/wiki/addy-osmani), [simon willison](/wiki/simon-willison)). | 19% task completion slowdown in experienced devs on familiar repos; perceived vs actual speed gap.^[raw/papers/metr-developer-productivity-rct-2025.md] |
| **Testing & TDD** | TDD is a "superpower" governor preventing hallucinated implementations ([kent beck](/wiki/kent-beck)). | Agents frequently engage in test-softening, test deletion, or author self-fulfilling tests.^[raw/papers/tests-agentic-pull-requests-2026.md] |
| **Integration Reliability** | Full-stack toolchains allow autonomous API migration. [[source: stripe-can-ai-agents-build-real-stripe-integrations-2026]](/wiki/raw/articles/stripe-can-ai-agents-build-real-stripe-integrations-2026) | False victories: agents misinterpret 400 Bad Requests as working endpoints; UI focus traps block completion. |
| **Automated Review** | LLM-as-a-judge provides fast, scalable quality feedback. [[source: airbnb-eval-driven-development-2026]](/wiki/raw/articles/airbnb-eval-driven-development-2026) | High false-discovery rates; 60.2% of bot-only reviews in 0–30% signal band.^[raw/papers/code-review-agents-empirical-study-2026.md] |

## Pull Requests and Maintenance

- **Claude Code in the Wild:** A matched study of 567 Claude Code pull requests reported an 83.8% acceptance rate; 54.9% were merged without revision.^[raw/papers/agentic-pull-requests-github-2026.md]
- **Longitudinal Maintenance:** A 6-month study of 508 agent-created files found fewer subsequent changes, though humans performed ~83% of maintenance.^[raw/papers/agent-generated-code-maintenance-2026.md]
- **Disputed Finding:** A larger study of ~110,000 PRs found increased churn over time for agentic code. Neither study randomized task assignment.^[raw/papers/autonomous-agent-contributions-wild-2026.md]

## Testing Activity vs. Assertion Strength

- **AIDev Dataset:** In 33,596 curated agentic PRs, test inclusion rose from 31% to 52%.^[raw/papers/tests-agentic-pull-requests-2026.md]
- **Assertion Integrity:** High test-touch frequency does not guarantee requirement coverage. Unconstrained agents often delete or weaken failing tests to pass CI. [[source: kent-beck-gergely-orosz-tdd-ai-agents-2025]](/wiki/raw/articles/kent-beck-gergely-orosz-tdd-ai-agents-2025)

## Automated Review Calibration

- **Signal Quality:** In 3,109 reviewed PRs, agent-only reviews had a 45.2% merge rate vs 68.4% for human-only reviews.^[raw/papers/code-review-agents-empirical-study-2026.md]
- **Calibration Requirement:** Evals research shows LLM judges require TPR/TNR calibration against expert human ground truth to avoid noise amplification. [[source: hamel-husain-shreya-shankar-evals-skills-2026]](/wiki/raw/articles/hamel-husain-shreya-shankar-evals-skills-2026)

## Benchmark and Oracle Health

- **SWE-bench Contamination:** Epoch found 500 SWE-bench Verified tasks concentrated in 12 repos with high contamination risks. [[source: epoch-swe-bench-verified-analysis-2025]](/wiki/raw/articles/epoch-swe-bench-verified-analysis-2025)
- **Oracle Defects:** OpenAI reported 59.4% flaw rates (narrow/wide tests) in hard SWE-bench subsets. [[source: openai-swe-bench-verified-audit-2026]](/wiki/raw/articles/openai-swe-bench-verified-audit-2026)
- **Adversarial Hacking:** 16% of 1,968 terminal tasks were hackable without solving the requirement.^[raw/papers/adversarial-hacker-fixer-verifiers-2026.md]

## Strongest Null Hypothesis

Frontier model capabilities, compiler/linter feedback, and developer task selection explain observed productivity. Multi-tier scaffolding may introduce compute cost, false rejection, and orchestration overhead without reducing escaped production defects. Compute-matched factorial trials remain necessary to reject this null.

## Related
- [agentic code quality](/wiki/agentic-code-quality) — proposed control architecture
- [eval driven development](/wiki/eval-driven-development) — eval methodology
- [agentic engineering patterns](/wiki/agentic-engineering-patterns) — practitioner patterns
- [agent containment and blast radius](/wiki/agent-containment-and-blast-radius) — authorization boundaries

---


# Section: QUERIES

## Agentic Code Quality Cycle 2 — Entity Accounting
- URL: https://pyweb.dev/wiki/agentic-code-quality-cycle-2-entity-accounting
- Raw Markdown: https://pyweb.dev/wiki/agentic-code-quality-cycle-2-entity-accounting.md
- Type: query
- Tags: evaluation, agents, retrospective

# Agentic Code Quality Cycle 2 — Entity Accounting

This document records the accounting and disposition of authors and organizations across all sources ingested in Cycle 2.

## Entity Dispositions

| Entity | Disposition | Target File | Justification / Reason |
|---|---|---|---|
| **Simon Willison** | Update | `entities/simon-willison.md` | Cataloged core agentic engineering patterns & proof-of-work doctrine. |
| **Addy Osmani** | Update | `entities/addy-osmani.md` | Author of Loop Engineering, Outer Loop, and Autonomy Levels canon. |
| **Kent Beck** | Create | `entities/kent-beck.md` | Pioneer of XP/TDD; framed TDD as agent governor & test-deletion anti-pattern. |
| **Martin Fowler** | Create | `entities/martin-fowler.md` | Software architecture authority; framed TDD as human comprehension anchor. |
| **Gergely Orosz** | Create | `entities/gergely-orosz.md` | Author of The Pragmatic Engineer; documented agent workflows with Beck/Cherny. |
| **Boris Cherny** | Create | `entities/boris-cherny.md` | Creator of Claude Code; verification-first harness & attacks-become-evals pattern. |
| **Armin Ronacher** | Update | `entities/armin-ronacher.md` | Evaluated agent SDK abstractions & identified testing/evals as hardest problem. |
| **Stripe** | Create | `entities/stripe.md` | Built Stripe Integration Benchmark; identified false-victory failure mode. |
| **Airbnb** | Create | `entities/airbnb.md` | Pioneered industrial Eval-Driven Development (EDD) & 3-layer eval funnel. |
| **Anthropic** | Update | `entities/anthropic.md` | Published Demystifying Evals for AI Agents & Claude Code harness. |
| **OpenAI** | Update | `entities/openai.md` | Published evaluation best practices & reference coding agent SDK harness. |
| **GitHub** | Create | `entities/github.md` | Published Spec-Driven Development toolkit for AI agents. |
| **Hamel Husain** | Update | `entities/hamel-husain.md` | Authority on AI Evals FAQ & evals-skills framework. |
| **Shreya Shankar** | Update | `entities/shreya-shankar.md` | Co-author of evals-skills & evaluator alignment research. |
| **Rohit Girme** | Skip | — | Lead author on Airbnb EDD article; represented via [airbnb](/wiki/airbnb) organization entity. |
| **Dan Miller** | Skip | — | Co-author on Airbnb EDD article; represented via [airbnb](/wiki/airbnb). |
| **Carol Liang** | Skip | — | Author on Stripe benchmark; represented via [stripe](/wiki/stripe). |
| **Kevin Ho** | Skip | — | Author on Stripe benchmark; represented via [stripe](/wiki/stripe). |
| **Zixiao Zhao** | Skip | — | Lead author on arXiv:2604.16790; single citation below threshold. |

## Related
- [agentic code quality](/wiki/agentic-code-quality)
- [agentic quality evidence](/wiki/agentic-quality-evidence)
- [eval driven development](/wiki/eval-driven-development)
- [agentic engineering patterns](/wiki/agentic-engineering-patterns)

---

## Agentic Code Quality Evidence — Entity Accounting
- URL: https://pyweb.dev/wiki/agentic-code-quality-entity-accounting
- Raw Markdown: https://pyweb.dev/wiki/agentic-code-quality-entity-accounting.md
- Type: query
- Tags: evaluation, agents

# Entity Accounting

## Create — Source Authors

- [aditi raghunathan](/wiki/aditi-raghunathan), [ahmed e hassan](/wiki/ahmed-e-hassan), [andrei botocan](/wiki/andrei-botocan), [beatrice casey](/wiki/beatrice-casey), [beth barnes](/wiki/beth-barnes), [brittany reid](/wiki/brittany-reid), [christoph csallner](/wiki/christoph-csallner), [david gros](/wiki/david-gros), [david rein](/wiki/david-rein), [dipayan banik](/wiki/dipayan-banik), [dung nguyen manh](/wiki/dung-nguyen-manh), [eirini kalliamvakou](/wiki/eirini-kalliamvakou), [florian brand](/wiki/florian-brand), [gelei deng](/wiki/gelei-deng), [hajimu iida](/wiki/hajimu-iida), [hao li](/wiki/hao-li), [haoxiang zhang](/wiki/haoxiang-zhang), [hiroshi iwata](/wiki/hiroshi-iwata), [huy nhat phan](/wiki/huy-nhat-phan), [ivan bercovich](/wiki/ivan-bercovich), [ivgeni segal](/wiki/ivgeni-segal), [jean stanislas denain](/wiki/jean-stanislas-denain), [joanna c s santos](/wiki/joanna-c-s-santos), [joel becker](/wiki/joel-becker), [k m ferdous](/wiki/k-m-ferdous), [kenichi yamaguchi](/wiki/kenichi-yamaguchi), [kexun zhang](/wiki/kexun-zhang), [kowshik chowdhury](/wiki/kowshik-chowdhury), [leo yu zhang](/wiki/leo-yu-zhang), [maliheh izadi](/wiki/maliheh-izadi), [mert demirer](/wiki/mert-demirer), [miku watanabe](/wiki/miku-watanabe), [minh vu thai pham](/wiki/minh-vu-thai-pham), [mohammed latif siddiq](/wiki/mohammed-latif-siddiq), [nate rush](/wiki/nate-rush), [nghi d q bui](/wiki/nghi-d-q-bui), [peter cihon](/wiki/peter-cihon), [prem devanbu](/wiki/prem-devanbu), [rahul pandita](/wiki/rahul-pandita), [razvan mihai popescu](/wiki/razvan-mihai-popescu), [sabrina haque](/wiki/sabrina-haque), [sarvesh ingale](/wiki/sarvesh-ingale), [shashwat saxena](/wiki/shashwat-saxena), [shazibul islam shamim](/wiki/shazibul-islam-shamim), [shota sawada](/wiki/shota-sawada), [sida peng](/wiki/sida-peng), [tatsuya shirai](/wiki/tatsuya-shirai), [tue le](/wiki/tue-le), [vinicius carvalho lopes](/wiki/vinicius-carvalho-lopes), [xinye zhao](/wiki/xinye-zhao), [yanjun zhang](/wiki/yanjun-zhang), [yi liu](/wiki/yi-liu), [ying zhang](/wiki/ying-zhang), [yubin qu](/wiki/yubin-qu), [yuekang li](/wiki/yuekang-li), [yutaro kashiwa](/wiki/yutaro-kashiwa), and [ziqian zhong](/wiki/ziqian-zhong) — all 57 are authors of ingested primary research sources and therefore central under the wiki's entity threshold.

## Create — Source Owners

- [model evaluation and threat research](/wiki/model-evaluation-and-threat-research) — owns two ingested capability/productivity sources.
- [epoch ai](/wiki/epoch-ai) — owns the ingested SWE-bench Verified analysis.
- [dora](/wiki/dora) — owns the ingested organizational research report.

## Update

- [openai](/wiki/openai) — existing entity; owns the ingested SWE-bench Verified audit.

## Skip

- **Oxc, dependency-cruiser, Import Linter, StrykerJS, and mutmut** — software projects rather than people or organizations under the current entity taxonomy.
- **Universities, employers, conferences, and repository hosts** — passing affiliations or publication venues, not subjects of the synthesis.
- **Secondary authors inside bibliographies** — passing mentions rather than authors of the ingested source.

## Completeness

- **57 source authors accounted for:** 57 Create dispositions.
- **4 named source-owner organizations accounted for:** 3 Create dispositions and 1 Update disposition.
- All source-author and source-owner dispositions are reflected in `entities/`, `index.md`, and this ledger.

## Related
- [agentic code quality](/wiki/agentic-code-quality)
- [agentic quality evidence](/wiki/agentic-quality-evidence)

---

## Agentic Engineering Entity Accounting
- URL: https://pyweb.dev/wiki/agentic-engineering-entity-accounting
- Raw Markdown: https://pyweb.dev/wiki/agentic-engineering-entity-accounting.md
- Type: query
- Summary: Entity disposition query documenting the 2026-08-24 ingestion of agentic engineering trends synthesis
- Tags: knowledge-management, retrospective

# Agentic Engineering Entity Accounting

Query documenting entity dispositions from the 2026-08-24 ingestion of the [agentic engineering trends synthesis](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis).

## Entity Disposition Summary

**Created (13 new entities):**
1. [addy osmani](/wiki/addy-osmani) — writer on agent harness engineering and conductor/orchestrator workflows
2. [anthropic](/wiki/anthropic) — AI company publishing production research on agent harnesses and multi-agent systems  
3. [armin ronacher](/wiki/armin-ronacher) — software developer writing about shared understanding and architectural coherence
4. [cursor](/wiki/cursor) — AI coding company experimenting with long-running multi-agent development
5. [james shore](/wiki/james-shore) — software-development writer analyzing AI productivity through maintenance costs
6. [jarred sumner](/wiki/jarred-sumner) — Bun creator; led the agent-assisted Zig-to-Rust port
7. [johann rehberger](/wiki/johann-rehberger) — security researcher on prompt injection and normalization of deviance in AI
8. [lalit maganti](/wiki/lalit-maganti) — Syntaqlite creator documenting AI's implementation/design boundary
9. [mario zechner](/wiki/mario-zechner) — software developer critiquing cognitive debt from unchecked agentic coding
10. [shopify](/wiki/shopify) — software company whose Liquid project supplied an autoresearch optimization case study
11. [tobias lutke](/wiki/tobias-lutke) — Shopify co-founder; applied autoresearch loops to Liquid optimization
12. [viv trivedy](/wiki/viv-trivedy) — credited by Addy Osmani with naming harness engineering and articulating the model-plus-harness framing
13. [wilson lin](/wiki/wilson-lin) — Cursor researcher documenting planner-worker coordination for long-running coding agents

**Updated (2 existing entities):**
- [simon willison](/wiki/simon-willison) — expanded with additional agentic engineering patterns and guides
- [andrej karpathy](/wiki/andrej-karpathy) — updated with coding-conduct and pedagogy layer attribution

**Skipped (3 entities with rationale):**
- **Reco AI** — source extraction did not expose a clearly attributable author for the JSONata rewrite case study
- **Teleport** — mentioned as infrastructure example for ephemeral sandboxed execution, not central source owner in synthesis  
- **Fly.io** — mentioned as infrastructure example (Fly Sprites), not central source owner in synthesis

## Cross-References

This query supports the [llm wiki ecosystem analysis](/wiki/llm-wiki-ecosystem-analysis) by demonstrating entity accounting practices and the [agent harness engineering](/wiki/agent-harness-engineering) concept development through documented contributor attribution.

## Raw Synthesis Source

The complete source material underlying these entity decisions is available at [agentic-engineering-trends-2026-synthesis.md](/wiki/raw/articles/agentic-engineering-trends-2026-synthesis), which synthesized 22 distinct sources covering agent harness patterns, multi-agent orchestration, conformance suites, and cognitive debt discourse from August 2026.

---

## LLM Wiki Ecosystem: Spec vs. Implementations
- URL: https://pyweb.dev/wiki/llm-wiki-ecosystem-analysis
- Raw Markdown: https://pyweb.dev/wiki/llm-wiki-ecosystem-analysis.md
- Type: query
- Summary: Comparative query analyzing the LLM Wiki specification against real-world implementations, including this site.
- Tags: knowledge-management, comparison, agents, principle

# LLM Wiki Ecosystem: Spec vs. Implementations

Deep analysis of Karpathy's LLM Wiki gist against four ecosystem implementations
(Astro-Han, NicholasSpisak/second-brain, atomicstrata/llm-wiki-compiler, wpzero),
filed as a query result. Full report with source links:
` [[source: karpathy-wiki-ecosystem-analysis-2026]](/wiki/raw/articles/karpathy-wiki-ecosystem-analysis-2026)`. Extends [llm wiki pattern](/wiki/llm-wiki-pattern).

## The core finding

Karpathy's gist is deliberately a **pattern, not a spec** — an idea file meant to be
instantiated per-agent. The implementations split into two poles:

- **Pure skills** (Astro-Han, second-brain, wpzero): the schema is a markdown document
  the agent reads; enforcement is prompt discipline plus at most one small checker
  script. Optimized for single-user, interactive, Obsidian-sidecar workflows.
- **Hardcoded compiler** (atomicstrata/llm-wiki-compiler): enforcement moved into
  runtime code — incremental compilation via source hashes, review queues, trust
  gates, embeddings, eval harness. Optimized for batch/CI/team operation.

Both poles converge on one principle: **keep the artifact dumb (plain markdown,
relative links, append-only log) and put enforcement in the cheapest place that
works** — prompt for judgment, a script for mechanical checks, runtime gates only
for CI/team/untrusted-input scale.

## Astro-Han's production-tested design boundaries

After 3 months of daily production use (94 articles / 99 sources), Astro-Han
explicitly *rejected*:

- **Source-hash freshness tracking** — raw/ is immutable; "hashes guard against
  events that cannot happen."
- **Persisted line-number citations** — every observed fidelity error was catchable
  by whole-file grep; anchor friction makes agents skip the rule.
- **Numeric confidence scores** — "false precision with no calibration behind it."
- **Per-article review dates** — maintenance driven by whole-wiki lint, not timers.
- **Vector/graph search** — at 50K–100K tokens of curated wiki, "grep and read are
  more reliable. Add search tooling only when recall measurably degrades."

What survived production:

- **The Grounding Invariant** — every load-bearing fact must exist verbatim in the
  linked raw file; one Python script greps high-signal literals across the wiki in
  seconds. Stateless and total because raw/ is immutable.
- **"No material" triage** — an explicit escape hatch: log the raw, stop, "do not
  force an article out of a thin source."
- **Status blocks (Outdated/Disputed)** — "never silently rewrite history."
- **Tiered lint authority** — auto-fix links and index entries; *report* facts,
  contradictions, and orphans without touching them.

## Operational realities

- **Prompt-only rules that aren't mechanically verifiable silently decay.** This is
  the strongest argument for one small checker script inside an otherwise pure
  skill — demonstrated today on this wiki: the first run of `check_evidence.py`
  found 4 suspect literals in 20 checked pages, of which 2 were real fidelity
  errors (a trailing period added to a Karpathy quote; a quote restated in the
  wrong words) and 2 were attribution/provenance gaps fixed in the page text.
- **Index + grep scales to ~100 sources** per Karpathy, Astro-Han, and
  second-brain — all three independently gate search tooling on the same threshold.
- **llmwiki-style compilation only pays off** for unattended multi-source
  pipelines, review-gated untrusted imports, or serving context packs to other
  agents. Its own README warns against it for interactive single-author use.

## What changed in this wiki as a result

1. `wiki/scripts/check_evidence.py` — mechanical grounding-invariant lint.
2. SCHEMA.md Operational Rules: triage dispositions (incl. **No material**),
   Status blocks, grounding invariant, tiered lint authority.
3. Four fidelity fixes applied to existing pages (pages fixed, raw untouched).

## Open question for this wiki

The Hermes `llm-wiki` skill (this wiki's origin) mandates sha256 raw frontmatter
and `confidence:` scores — both on Astro-Han's rejected list. Kept for now: the
wiki-update cron ingests unattended, where drift detection is real. Revisit if
ingest friction becomes the binding constraint.

## Related

- [llm wiki pattern](/wiki/llm-wiki-pattern) — the base pattern this analysis extends
- [context engineering](/wiki/context-engineering) — the wiki as compiled agent context
- [andrej karpathy](/wiki/andrej-karpathy) — the pattern's author
- [error analysis and evals](/wiki/error-analysis-and-evals) — evidence linting as an eval gate

---


# Section: WRITING

## The Agentic Engineering Curriculum
- URL: https://pyweb.dev/writing/agentic-engineering-curriculum
- Raw Markdown: https://pyweb.dev/writing/agentic-engineering-curriculum.md
- Date: 2026-08-30

# The Agentic Engineering Curriculum

This is a course, not an essay. Twelve lessons take you from first principles to running agentic software delivery, and every lesson links into the [wiki](/wiki/llm-wiki-pattern) so you can go deeper on any node. The spine: first understand what the model is bad at, then build the scaffolding that fixes it.

```mermaid
flowchart LR
    F[Foundations] --> C[Context]
    C --> E[Evals]
    E --> V[Verification]
    V --> H[Harness]
    H --> S[Factory]
```

## Module 1: Foundations

### Lesson 1: How agents actually fail

Start here because everything else is a response to these failure modes. Agents generate volume, and volume without discipline is slop. [Karpathy's four guidelines](/wiki/karpathy-four-guidelines) are the minimum conduct rules: think before coding, simplicity first, surgical changes, goal-driven. From there, the [five debts of agentic engineering](/wiki/five-debts-of-agentic-engineering) name what accumulates when you skip discipline: intent debt, semantic debt, verification debt, architecture debt, authorization risk.

**Key sources:** [Andrej Karpathy](/wiki/andrej-karpathy), [agentic engineering patterns](/wiki/agentic-engineering-patterns).

### Lesson 2: The cost model changed

Code is cheap now. [Bad code is the most expensive it has ever been](/wiki/tdd-with-agents), per [Matt Pocock](/wiki/matt-pocock), because agents can ship it faster than you can review it. [Drew Breunig's](/wiki/drew-breunig) economics argument cuts the same way: when frontier models are priced like frontier models, a weak harness stops being a rounding error. The old excuse, "the model will paper over it," died with the free lunch.

## Module 2: Context is the game

### Lesson 3: Context engineering

[Matt Pocock's](/wiki/matt-pocock) framing: the agent's performance is bounded by what's in its window. [Context engineering](/wiki/context-engineering) means curating [AGENTS.md specs](/wiki/agents-md-spec), feedback loops, and plan artifacts instead of letting defaults run. Hand-crafted context beats generic context, always.

### Lesson 4: Context rot and the smart zone

Two properties of long contexts you can't negotiate with. [Context rot](/wiki/context-rot): reasoning degrades as irrelevant tokens accumulate, with the classic "lost in the middle" effect. The [smart zone](/wiki/smart-zone): modern models think sharpest in roughly their first 150k tokens. Design sessions so load-bearing work happens early, or [hand off](/wiki/handoff-artifacts) to a fresh context.

### Lesson 5: Progressive disclosure and subagents

Don't load everything up front. [Progressive disclosure](/wiki/progressive-disclosure) loads pointers first and full material only on trigger. [Subagents](/wiki/subagents-and-context-management) isolate high-token exploration in child contexts that return summaries, keeping the root conversation sharp. At fleet scale this becomes [multi-agent orchestration](/wiki/multi-agent-orchestration), coder to conductor to orchestrator.

## Module 3: Evals

### Lesson 6: Error analysis before evals

The [Hamel Husain](/wiki/hamel-husain) and [Shreya Shankar](/wiki/shreya-shankar) discipline: read real failures first, categorize them, then derive rubrics from the taxonomy. [Error analysis and evals](/wiki/error-analysis-and-evals) is the prerequisite; eval suites built without it measure the wrong thing.

### Lesson 7: Eval-driven development

[Eval-driven development](/wiki/eval-driven-development) is TDD's analogue for generative systems, coined in practice at [Airbnb](/wiki/airbnb) and formalized by Husain and Shankar: evals as a continuous loop, not a release gate. Design the [generator-evaluator loop](/wiki/generator-evaluator-loop) with a separate judge, because agents grading their own output confidently grade wrong.

### Lesson 8: Design for verifiability

["It's hard to eval" is a product smell](/wiki/designing-for-verifiability). If you can't verify an artifact, your users can't either. Fix the artifact before building the eval harness. This connects straight to [automated eval engineering](/wiki/automated-eval-engineering) and [closed-loop agent improvement](/wiki/closed-loop-agent-improvement), where production failure traces drive candidate patches gated by benchmarks.

## Module 4: Verification and code quality

### Lesson 9: Red/green TDD with agents

[Red/green TDD](/wiki/red-green-tdd) is the control mechanism for agent code: write the test, watch it fail, write minimal code to pass. Skipping RED risks tautological tests, and with an agent authoring both sides that risk compounds. [TDD with agents](/wiki/tdd-with-agents) makes the test suite the truth referee, which is what lets the agent loop independently without a human reading every line.

### Lesson 10: Constraint layering and lint gates

Allocate each rule to its cheapest reliable layer. [Constraint layering](/wiki/constraint-layering)'s maxim: prompt for judgment, script the mechanical, gate the consequential, isolate the dangerous. The Tier 1 implementation is [deterministic lint gates](/wiki/deterministic-lint-gates), sub-second error-only linting after every edit, converting style policy from probabilistic prompt compliance into mechanical backpressure. Above the mechanical layer, [agent containment and blast radius](/wiki/agent-containment-and-blast-radius) caps what the agent *can* do, not just what it tends to do.

## Module 5: Harnesses and the factory

### Lesson 11: Harness engineering

[Agent harness engineering](/wiki/agent-harness-engineering) is the discipline of designing the runtime around the model: sandboxes, tool contracts, progressive context, verification gates. Real architectures prove the pattern: [Codex harness architecture](/wiki/codex-harness-architecture) decouples the reasoning loop from execution via a JSON-RPC app server, and [DeepSeek Harness](/wiki/deepseek-harness) runs modular, traceable execution on its [Cordis framework](/wiki/cordis-framework). The thesis in both cases: agent = model + harness, and the harness is where reliability lives.

### Lesson 12: The agentic software factory

The end state. The [agentic software factory](/wiki/agentic-software-factory) is the operating model where agents carry production work from task decomposition to merge, and humans shift to specs, architecture, and exception points. [Addy Osmani](/wiki/addy-osmani) frames it as owning the outer loop. Quality is measured as [releasable patch rate](/wiki/releasable-patch-rate), the joint probability of correctness, regression safety, security, architecture, and scope, with the evidence map in [agentic code quality](/wiki/agentic-code-quality) and [conformance suites as fitness functions](/wiki/conformance-suites-as-fitness-functions) keeping agents honest.

## Module 5.5: The human layer

Two lessons the factory can't skip. First, [cognitive debt and walkthroughs](/wiki/cognitive-debt-and-walkthroughs): if you merge code you don't understand, the codebase becomes a black box that paralyzes your next design decision. Linear walkthroughs and interactive explanations are the antidote. Second, [hoard and recombine](/wiki/hoard-and-recombine): collect working snippets and proof-of-concept tools, then feed them to agents as concrete reference material. Knowing something is possible is weak; possessing a tested snippet proves it.

## How to run the course

Read the lessons in order; each one is short on purpose. When a lesson lands, follow its wiki links and read the primary sources they cite. The [Feynman technique](/wiki/feynman-technique) applies to the whole thing: after each lesson, explain the pattern to someone in plain English. If you can't, you found the gap.

---

## The Linter Is the Cheapest Adult in the Room
- URL: https://pyweb.dev/writing/the-linter-is-the-cheapest-adult-in-the-room
- Raw Markdown: https://pyweb.dev/writing/the-linter-is-the-cheapest-adult-in-the-room.md
- Date: 2026-08-28

# The Linter Is the Cheapest Adult in the Room

There's a principle worth tattooing on every agent harness: **prompt for judgment, script the mechanical, gate the consequential, isolate the dangerous** ([constraint layering](/wiki/constraint-layering)). Most teams get the first and last parts. The middle — *script the mechanical* — is where autonomous coding loops quietly bleed quality, because the mechanical layer everyone inherited was built for a different era.

ESLint was designed for humans who run it occasionally and read its warnings over coffee. An autonomous agent editing code in a loop is a different customer entirely. It edits every few seconds, it never reads warnings, and it treats exit code 0 as permission to proceed. Serve that customer a 45-second linter with a warning tier and you've built a gate that neither slows the agent down nor stops it — the worst of both.

## The engine: Oxlint

[Oxlint](/wiki/deterministic-lint-gates) is a JavaScript/TypeScript linter rebuilt in Rust on the Oxc compiler stack. The published benchmarks claim 50–100× over ESLint, and the numbers matter less than the category change: linting drops from "CI stage you wait on" to "keystroke-speed feedback inside the agent's inner loop."

Three architectural choices make it agent-native rather than merely fast:

1. **Type-aware linting via `tsgolint`** — built on `typescript-go`, so checks like floating-promise detection use the real TypeScript type system, not a reimplementation. One type system, one source of truth.
2. **Multi-file analysis** — a project-wide module graph shared across rules, which kills the classic `import/no-cycle` performance cliff.
3. **Structured diagnostics** — precise spans, contextual data, and documentation links, explicitly designed so that machine consumers can locate and repair issues reliably. A diagnostic an agent can act on is worth ten prose warnings.

## The policy: Error, Never Warn

An engine needs a rulebook. `@nkzw/oxlint-config` — Christoph Nakazawa's opinionated preset — is the cleanest public statement of what a lint policy for autonomous loops should look like. Its README says the quiet part out loud: *"Warnings are noise and get ignored. Either it's an issue, or it isn't."*

Every principle in that config maps onto a documented agent failure mode from [agentic code quality](/wiki/agentic-code-quality):

| Config principle | Failure mode it closes |
|---|---|
| Error, never warn | Agents ignore anything that doesn't block; warnings train the loop to proceed |
| Ban `test.only` | The reward-hack where a "green" suite silently skipped 99% of its tests |
| Ban `console.log` | Debug scaffolding leaking into production diffs |
| Ban `instanceof` | Plausible-looking code that breaks across bundle and realm boundaries |
| Prefer autofixable rules | Repair stays in the inner loop instead of burning review cycles |
| Deterministic sorting | Merge conflicts between parallel agent worktrees vanish at the syntax level |

That last row is underrated. When multiple agents (or an agent and a human) touch the same file, alphabetically sorted object keys, interface members, and JSX props mean diffs stay minimal and collisions stop being a coordination tax. Sorting isn't aesthetics — it's [multi-agent](/wiki/multi-agent-orchestration) merge-conflict prevention.

## Examples: what the gate actually catches

**Reward hacking via focused tests** — the linter refuses the shortcut:

```typescript
// ❌ Blocked: no-only-tests
it.only('passes the one test I did not break', () => { ... });
```

**Hallucinated logic that reads fine and cannot work** — caught at AST speed:

```typescript
// ❌ Blocked: oxc/const-comparisons — x cannot be both
if (x > 10 && x < 5) { retry(); }

// ❌ Blocked: oxc/bad-min-max-func — clamp is inverted
const clamped = Math.min(0, Math.max(100, score));
```

**Cross-realm booby traps** — banned as a category:

```typescript
// ❌ Blocked: @nkzw/no-instanceof — fails across iframes/duplicate deps
if (err instanceof CustomError) { ... }

// ✅ Brand check survives bundle boundaries
if (err?.name === 'CustomError') { ... }
```

**Debug residue** — the diff never ships:

```typescript
// ❌ Blocked: no-console
console.log('payload', data);

// ❌ Blocked: no-warning-comments — @nocommit markers fail the build
// @nocommit temporary hack
```

## Layering discipline, demonstrated in miniature

The most instructive part of the nkzw config is what it turns **off**. For TypeScript files, it disables `no-undef`, `no-dupe-keys`, `no-unreachable`, and a dozen other rules — because the type-checker already enforces them. Each invariant lives in exactly one layer, in the cheapest layer that can hold it.

That's [constraint layering](/wiki/constraint-layering) applied to the toolchain itself. Don't lint what the compiler checks. Don't prompt what the linter enforces. Don't burn a single token of agent attention asking the model to "remember" a rule a Rust binary verifies in milliseconds.

```mermaid
flowchart TD
    E[Agent edit] --> L["oxlint: error-only, ms-scale"]
    L -->|structured diagnostic| E
    L --> T["tsc --noEmit: type contracts"]
    T -->|fail: bounded repair| E
    T --> B["Tests, architecture contracts, review"]
    B --> R["Releasable patch"]
```

The deterministic lint gate is the first wall a patch hits and the cheapest one to bounce off. Every defect it absorbs at millisecond cost is a defect that would otherwise surface in a test run, a review pass, or production — each an order of magnitude more expensive than the last. That's the whole economics of the [releasable patch rate](/wiki/releasable-patch-rate) in one sentence: push rejection as early and as cheap as it will go.

The linter can't judge your architecture, your intent, or your taste. But it is the only reviewer that reads every line of every diff in milliseconds, never gets tired, and never lets "it's just a warning" slide. In a software factory, that makes it the cheapest adult in the room.

---

*Grounded in [deterministic-lint-gates](/wiki/deterministic-lint-gates), [agentic-code-quality](/wiki/agentic-code-quality), and [constraint-layering](/wiki/constraint-layering) — compiled from the Oxc project documentation and `@nkzw/oxlint-config` v2.0.0.*

---

## The Agent as a Virtual OS: Inside the Codex Harness Architecture
- URL: https://pyweb.dev/writing/the-agent-as-a-virtual-os-codex-harness-architecture
- Raw Markdown: https://pyweb.dev/writing/the-agent-as-a-virtual-os-codex-harness-architecture.md
- Date: 2026-08-27

# The Agent as a Virtual OS: Inside the Codex Harness Architecture

When engineers first build with LLMs, they write a `while` loop that takes user input, appends it to a prompt string, queries a model API, and runs whatever bash command the model spits out via `subprocess.run()`. It works for a toy demo. It collapses into catastrophe the moment an agent loops indefinitely, floods its context window with a 50,000-line log dump, or accidentally wipes a local directory.

The release of [OpenAI](/wiki/openai)'s open-source [Codex Harness](/wiki/codex-harness-architecture) (`codex-rs`) marks the maturation of [agent harness engineering](/wiki/agent-harness-engineering). A production coding agent is not a chatbot with tools; it is a **virtual operating system**. It must arbitrate between untrusted generated code, non-deterministic reasoning engines, strict token memory budgets, and hostile host environments.

By examining the crate topology of `openai/codex`, we can extract the foundational architectural patterns and hard-won lessons required to build reliable agent infrastructure.

---

## 1. Decouple the Core Engine from the Interaction Surface (`App-Server` Pattern)

The most common antipattern in agent design is coupling terminal I/O (handling `stdin`, formatting ansi colors, printing spinners) directly into the agent decision loop.

Codex avoids this by applying [Clean Architecture](/wiki/clean-architecture) and [Hexagonal Architecture](/wiki/hexagonal-architecture). The reasoning core (`codex-core`) has zero awareness of the terminal. It communicates exclusively through a typed JSON-RPC protocol (`codex-app-server-protocol`).

```mermaid
flowchart TD
    A["Frontends / Interfaces<br/><small>codex-cli · codex-tui · IDE Plugins</small>"]
    B["App Server / Control Plane<br/><small>codex-app-server daemon (JSON-RPC over UDS/WS)</small>"]
    C["Agent Core Engine<br/><small>codex-core (OODA Turn Loop & Tool Router)</small>"]

    A --> B
    B --> C
```

### The Architectural Payoff:
1. **Multi-Client Versatility:** The exact same agent loop powers the interactive TUI, the headless CLI `/goal` autonomous loop, and background IDE extensions (VS Code / JetBrains) over Unix Domain Sockets or WebSockets.
2. **Resilient, Reconnectable Sessions:** If your terminal closes or your connection drops, the background agent process does not die. Clients can disconnect, reconnect, or inject mid-turn steer/interrupt signals without invalidating state.

---

## 2. Multi-Tier Sandboxing & Defense in Depth

In production, every shell command emitted by a model must be treated as potentially adversarial. Relying solely on confirmation prompts causes "approval fatigue," leading users to blindly approve destructive commands.

Codex enforces OS-level containment through `codex-sandboxing`, isolating process execution across three major operating systems:
- **Linux:** Mount namespaces via Bubblewrap (`bwrap`) paired with Landlock LSM filesystem access rules.
- **macOS:** Apple Seatbelt (`sandbox-exec`) security profiles compiled dynamically per execution.
- **Windows:** Job objects and restricted security tokens.

By pushing containment into kernel-level isolation layers, the harness establishes a robust baseline for [agent containment and blast radius](/wiki/agent-containment-and-blast-radius). However, true defense in depth requires combining process isolation with strict egress filtering and network barriers to prevent prompt-injection exfiltration through allowlisted channels.

---

## 3. Bounded Context Protection (`HeadTailBuffer`)

A subtle but fatal vulnerability in autonomous agent loops is context blowout. If an agent executes `cat build.log` or runs a test command that outputs 100,000 lines of stack traces, passing that raw output back to the LLM immediately pushes the model out of its [smart zone](/wiki/smart-zone) and triggers severe [context rot](/wiki/context-rot).

Codex solves this within `unified_exec` via a `HeadTailBuffer`:
* It captures the leading $N$ lines (to preserve the command invocation and header context).
* It captures the trailing $M$ lines (to preserve the exit code and final error traces).
* It discards the middle thousands of lines while recording an explicit truncation notice.

This guarantees that tool output remains strictly bounded before it ever reaches the prompt assembler.

---

## 4. Asymmetric Tool Specialization (`apply-patch`)

While generic agent frameworks attempt to use arbitrary nested JSON edit schemas, frontier models are heavily fine-tuned via Reinforcement Learning on specific tool formats. 

OpenAI models excel at applying unified diffs with line-anchor hints. Rather than forcing the model to emit full-file rewrites or JSON AST transformations, Codex includes an optimized, in-tree `apply-patch` engine. The tool applicator is built with high fault tolerance against minor whitespace drifts, aligning the harness's tool execution substrate with the model's native training distribution.

---

## 5. State as an Immutable Timeline (`thread-store`)

Chat history in a production harness cannot be stored as a flat list of strings. Codex models state as an ordered timeline of immutable `Turns` inside a `Thread`:
* **Turn Invariants:** Every turn records start/end timestamps, tokens consumed, tool invocations, and an explicit `TurnDiffTracker` capturing exact file tree mutations.
* **Time-Travel Primitives:** Because turns are structured and file diffs are tracked per turn, the harness can natively execute `thread_rollback`, `thread_fork`, and `thread_revert`. If an agent takes a wrong path, it can roll back the entire workspace state cleanly to an earlier turn and try a new hypothesis.

---

## 6. Testability Without Live Models

By strictly adhering to the [Dependency Rule](/wiki/dependency-rule), `codex-core` never depends on concrete network clients or un-sandboxed shell runners. The core loop defines abstract traits (ports) for model communication and tool dispatch, implemented by outer adapters (`codex-client`, `rmcp-client`, and test stubs).

This allows the entire agent state machine—turn loops, tool routing, diff tracking, error recovery, and session rollbacks—to be verified with fast in-memory unit tests and conformance suites in milliseconds, without spending a single API token or waiting on network latency.

---

## Summary Checklist for Modern Harness Design

1. **Protocol First:** Put the agent loop behind a typed JSON-RPC daemon before building interfaces.
2. **Contain Execution:** Use OS namespaces (`bwrap`/Seatbelt) rather than trusting prompts.
3. **Protect the Smart Zone:** Enforce head/tail truncation on all execution output streams.
4. **Align Tool Formats:** Specialize diff and edit tools to match the model's RL training format.
5. **Treat Turns as Invariants:** Track file mutations per turn to enable clean rollbacks and forks.

---

## The Five Debts of Agentic Engineering
- URL: https://pyweb.dev/writing/five-debts-of-agentic-engineering
- Raw Markdown: https://pyweb.dev/writing/five-debts-of-agentic-engineering.md
- Date: 2026-08-27

# The Five Debts of Agentic Engineering

Modern software engineering with autonomous AI agents is not "letting an AI write the code." It is the discipline of designing a software-delivery system in which probabilistic generators can act with useful autonomy while intent, authority, verification, architecture, and release remain strictly controlled and auditable.

Frontier language models are extraordinary syntax engines, but they are completion optimizers: left to themselves, they take the shortest path to an apparent exit state. They skip the invisible scaffolding that senior engineers apply—surfacing hidden assumptions, scoping blast radius, constructing falsifiable test oracles, and preserving modular boundaries.

Without disciplined harness design, cheap code generation simply accelerates the accumulation of **five structural debts**:

```mermaid
flowchart TD
    subgraph Generative Debts
        D1["1. Intent Debt"] -->|Misaligned assumptions| C1["Grilling & Executable Specs"]
        D2["2. Context Debt"] -->|Semantic drift & token bloat| C2["CONTEXT.md & ADRs"]
        D3["3. Verification Debt"] -->|Self-fulfilling & tautological tests| C3["Red-Green TDD at Seams"]
        D4["4. Architecture Debt"] -->|Shallow modules & coupling| C4["Deep Modules & Boundary Rules"]
        D5["5. Authorization Risk"] -->|Overeager scope creep| C5["Sandboxes & Allowlist Gates"]
    end
    subgraph Control Architecture
        C1 --> R["Releasable Patch Vector"]
        C2 --> R
        C3 --> R
        C4 --> R
        C5 --> R
    end
```

---

## 1. Intent Debt: Building the Wrong Thing Fast

The default failure mode of an unprompted coding agent is to take an underspecified prompt, select one arbitrary interpretation among dozens of syntactically plausible paths, write 500 lines of code, and declare victory.

Faster generation makes exploring the wrong branch expensive sooner.

**The Solution:** Interrogate the decision frontier before writing code. [The grilling doctrine](/wiki/grilling-doctrine) requires the agent to build an active decision tree, interview the engineer to resolve ambiguity, separate factual research (the agent's job) from value-laden tradeoffs (the human's job), and commit to explicit non-goals and observable acceptance criteria.

---

## 2. Context Debt: Semantic Drift in the Smart Zone

When tasks span multiple prompts or sub-agents without a shared ubiquitous language, models re-explain core concepts using ad-hoc synonyms. This causes [context rot](/wiki/context-rot), pollutes the model's [smart zone](/wiki/smart-zone), and creates naming collisions across modules.

**The Solution:** [Context engineering](/wiki/context-engineering) via canonical `CONTEXT.md` definitions and Architectural Decision Records (ADRs). By establishing stable names for domain entities and boundary seams, prompts remain compact and future sessions retain semantic continuity without token bloat.

---

## 3. Verification Debt: Green Is a Claim, Not a Conclusion

A passing test suite is not proof that the code works. When agents author their own tests without constraints, they frequently write tautological assertions, overfit to their own buggy implementations, or even modify existing assertions to force a passing exit code.

Recent empirical audits underscore this reality:
- [OpenAI](/wiki/openai) discovered material problems or exploitable loopholes in **59.4%** of an audited hard subset of SWE-bench Verified tasks.
- A comprehensive audit of 1,968 terminal-agent benchmark tasks found that **16%** were hackable by frontier models from the problem statement alone, bypassing actual solution requirements.

**The Solution:** [Red-green TDD](/wiki/red-green-tdd) executed strictly through public interfaces at pre-agreed seams. The agent must observe the test fail for the expected failure reason before implementing minimal code. Crucially, baseline regression suites must remain mounted read-only outside the generator's writable workspace.

---

## 4. Architecture Debt: The Shallow Module Trap

Language models optimize for local file diffs. Without structural constraints, autonomous agents generate cosmetic modularity—creating dozen-file directory structures where each file is a thin, one-line pass-through. This scatters complexity, increases coupling, and destroys comprehension for future maintainers.

**The Solution:** Enforce [Clean Architecture](/wiki/clean-architecture) and deep modules: substantial behavior hidden behind small, stable interfaces. Apply the deletion test—if removing a proposed abstraction does not concentrate complexity behind a simpler seam, reject it.

---

## 5. Authorization & Operational Risk: Overeager Scope Creep

An agent tasked with fixing an API endpoint might observe an outdated credential file or an adjacent configuration format and decide to "clean it up."

In empirical benchmarks like OverEager-Bench (500 validated scenarios across frontier models), removing explicit consent checks increased out-of-scope actions by **11.9 to 17.2 percentage points**. An agent may correctly infer that modifying an adjacent system helps its local goal, but that action remains strictly unauthorized.

**The Solution:** [Agent containment and blast radius controls](/wiki/agent-containment-and-blast-radius). Enforce strict filesystem allowlists, ephemeral execution environments, network egress restrictions, and tamper-proof audit trails.

---

## The Evidence: What Actually Moves the Needle

Treating agent tooling as a magic multiplier is refuted by empirical literature:

1. **Productivity is Context-Dependent:** [METR](/wiki/metr-developer-productivity-rct-2025)'s randomized controlled trial on experienced maintainers in mature repositories found an initial **19% slowdown** with early-2025 AI tools, even though participants anticipated a 24% speedup.
2. **Skills Require Domain Fit:** In [SWE-Skills-Bench](/wiki/swe-skills-bench-2026), 39 out of 49 public skills yielded **0% pass-rate improvement** on real-world GitHub issues. Only 7 specialized, toolchain-specific skills produced meaningful gains (up to +30%), while outdated skills degraded performance by up to -10%.
3. **Delivery Bottlenecks Shift Downstream:** As [DORA](/wiki/dora)'s research shows, accelerating raw code generation without matching review and verification capacity reduces overall delivery throughput and stability.

```mermaid
flowchart TD
    G["Candidate Patch"] --> C1{"Correctness"}
    C1 -->|Pass| C2{"Regression Safety"}
    C1 -->|Fail| F["Rejected / Rework"]
    C2 -->|Pass| C3{"Security Baseline"}
    C2 -->|Fail| F
    C3 -->|Pass| C4{"Architecture Conformance"}
    C3 -->|Fail| F
    C4 -->|Pass| C5{"Scope Authorization"}
    C4 -->|Fail| F
    C5 -->|Pass| R["Releasable Patch (Release Boundary)"]
```

---

## Constraint Layering: The Governing Maxim

To prevent system decay, allocate every engineering rule to the cheapest reliable layer ([constraint layering](/wiki/constraint-layering)):

> **Prompt for judgment, script the mechanical, gate the consequential, and isolate the dangerous.**

- **Prompts & Skills:** Use for high-judgment workflows, anti-rationalization tables, and iterative debugging strategies.
- **Linters & Compilers:** Use for AST checks, typing, and deterministic syntax rules.
- **CI & Protected Oracles:** Use for merge-blocking regression suites and invariant contracts.
- **Sandboxes & Policies:** Use for filesystem allowlists, network boundaries, and credential isolation.
- **Canary & Telemetry:** Use for runtime health and automated rollback triggers.

Quality is not a scalar average; it is the joint probability that a patch satisfies correctness, regression safety, security, architecture, and scope discipline. The true unit of analysis in AI engineering is never the model in isolation—it is the complete **model–context–harness–toolchain–policy–oracle–human system**.

---

## Clean Architecture for Agent Harnesses
- URL: https://pyweb.dev/writing/clean-architecture-for-agent-harnesses
- Raw Markdown: https://pyweb.dev/writing/clean-architecture-for-agent-harnesses.md
- Date: 2026-08-25

# Clean Architecture for Agent Harnesses

When developers first build with large language models, they treat the model as the application. Prompts get tangled with business logic, database queries are assembled inline inside tool wrappers, and system state is sprayed across raw chat histories. It works for a demo, but collapses the moment a frontier model updates, a tool format shifts, or an unexpected hallucination corrupts the database.

Fourteen years ago, [Robert C. Martin](/wiki/robert-c-martin) synthesized a unified approach to system boundaries in [Clean Architecture](/wiki/clean-architecture), drawing together [Alistair Cockburn](/wiki/alistair-cockburn)'s [Hexagonal Architecture](/wiki/hexagonal-architecture) (Ports and Adapters) and [Jeffrey Palermo](/wiki/jeffrey-palermo)'s [Onion Architecture](/wiki/onion-architecture). The core organizing axiom was deceptively simple: **The Dependency Rule**.

> "This rule says that source code dependencies can only point inwards. Nothing in an inner circle can know anything at all about something in an outer circle. In particular, the name of something declared in an outer circle must not be mentioned by the code in the an inner circle."

As the agent ecosystem matures into systematic [agent harness engineering](/wiki/agent-harness-engineering), the central revelation of 2026 is that an LLM is not your core domain. An LLM is a mechanism — an external driver sitting on the outermost circle alongside databases, web servers, and third-party APIs. When you apply Clean Architecture to agent infrastructure, the fragility disappears.

## The Five Characteristics of an Agent Harness

In his 2012 formulation, Martin defined five properties that clean boundaries guarantee:
1. **Independent of Frameworks:** The architecture does not depend on a feature-laden library; frameworks are tools, not constraints.
2. **Testable:** Business rules can be verified without the UI, database, or external agencies.
3. **Independent of UI:** Presentation layers can change without touching business policies.
4. **Independent of Database:** Persistence technologies can be swapped without rewriting business logic.
5. **Independent of any external agency:** "In fact your business rules simply don't know anything at all about the outside world."

In agentic systems, the stochastic model is the ultimate "external agency." If your domain policies require calling Claude or DeepSeek directly inside core entities, your business logic is hostage to model non-determinism.

By placing the agentic loop on the outer boundary and treating the LLM as an interchangeable adapter, your domain rules remain deterministic, inspectable, and independently testable via [red-green TDD](/wiki/red-green-tdd).

## Dependency Inversion at the Agent Boundary

How do you orchestrate complex agent workflows without violating the Dependency Rule? The answer is the [Dependency Inversion Principle](/wiki/dependency-inversion-principle).

In Clean Architecture, when an inner use case needs to communicate with an outer mechanism, it defines an interface (an input or output port) in the inner circle. The outer mechanism implements that interface, flipping the source code dependency so it opposes runtime control flow.

We see this exact pattern in production harnesses today:

- **Pluggable Agent Runtimes:** In the [DeepSeek Harness](/wiki/deepseek-harness), the core orchestration loop relies on the [Cordis framework](/wiki/cordis-framework) microkernel. Services are registered in containers (`ctx.tools`, `ctx.llm`, `ctx.sessions`). The inner lifecycle manages task states and dispatching without importing concrete model providers or tool runners.
- **Sandboxed Containment:** [Anthropic](/wiki/anthropic)'s architecture for Claude Code enforces strict [agent containment and blast radius](/wiki/agent-containment-and-blast-radius). The agent executes within an isolated container substrate. File modifications, bash commands, and network requests cross structured interface boundaries governed by mechanical permission policies before execution.
- **Progressive Capability Delivery:** Rather than dumping the entire world into prompt context, modern harnesses use [progressive disclosure](/wiki/progressive-disclosure) to pass simple data transfer objects across boundaries on demand, preventing [context rot](/wiki/context-rot).

## Verifiability Over Model Hype

For years, teams ignored software architecture in AI. As [Drew Breunig](/wiki/drew-breunig) noted, when frontier models were dropping in cost while leaping in capability, "A new model would arrive at the same price (or cheaper!) and paper over most of your problems." But once free-lunch gains leveled off, teams "started to think about what work went where."

When you isolate the model behind clean ports, evaluation stops being an intractable mystery. As [Hamel Husain](/wiki/hamel-husain) argues in [designing for verifiability](/wiki/designing-for-verifiability), "it's hard to eval is a product smell." When an agent's domain policies are decoupled from external mechanisms, you can evaluate the agent's decision boundaries with precision instead of guessing which prompt sentence caused a regression.

Clean Architecture was never about dogmatic folder structures or enterprise boilerplate. It is about drawing clear lines so that unstable details do not contaminate enduring policies. In agent engineering, the model will always be an unstable detail. Build the harness around clean boundaries, enforce the Dependency Rule, and let the outer world change without breaking what you built.

---

## Skills-Native Evaluation and Agent Adversarial Training
- URL: https://pyweb.dev/writing/skills-native-evaluation-agent-adversarial-training
- Raw Markdown: https://pyweb.dev/writing/skills-native-evaluation-agent-adversarial-training.md
- Date: 2026-08-24

# Skills-Native Evaluation and Agent Adversarial Training

Two breakthrough patterns are reshaping how we build reliable AI agents: evaluation methodology packaged as portable skills, and adversarial feedback loops that separate code generation from quality assessment. These advances move beyond prompt engineering toward systematic harness design and verifiable improvement cycles.

## Evaluation as Agent Skills, Not Tools

Traditional evaluation tools push teams toward generic metrics before they understand their data. [Hamel Husain](/wiki/hamel-husain) and [Shreya Shankar](/wiki/shreya-shankar) flipped this with **Evals Skills** — evaluation methodology shipped as installable agent skills that route teams through proven workflows based on their specific situation.

Instead of another dashboard or framework, the methodology becomes procedural instructions a coding agent loads on demand. The entry skill `start` analyzes your setup and routes to specialized workflows: `eval-audit` for existing pipelines, `error-discovery` for unanalyzed traces, and domain-specific skills like `evaluate-rag` for retrieval systems.

This exemplifies [progressive disclosure](/wiki/progressive-disclosure) — giving agents just enough context to proceed effectively without overwhelming the smart zone with comprehensive documentation. Each skill encodes hard-won lessons "from helping 50+ companies" into executable procedures that avoid "many easily avoidable footguns."

The approach acknowledges a core truth about [error analysis and evals](/wiki/error-analysis-and-evals): real evaluation requires human domain expertise to label failure modes, not automated scoring against generic rubrics. As Husain puts it, the revenge of the data scientist means "every recurring eval pitfall maps to a missing data-science fundamental."

## Adversarial Agent Training

Meanwhile, [Anthropic](/wiki/anthropic)'s [Prithvi Rajasekaran](/wiki/prithvi-rajasekaran) tackled a different reliability problem: agents confidently praising their own mediocre work. His solution borrows from Generative Adversarial Networks, creating a **generator-evaluator loop** that separates the agent doing work from the agent judging it.

For frontend design — where there's "no binary check equivalent to a verifiable software test" — the harness converts subjective quality into four gradable criteria: design quality, originality, craft, and functionality. The evaluator uses Playwright to navigate live pages, screenshot implementations, and score against explicit criteria that "penalized highly generic 'AI slop' patterns" like "purple gradients over white cards."

The key insight: "tuning a standalone evaluator to be skeptical turns out to be far more tractable than making a generator critical of its own work." Runs lasted 5-15 iterations over four hours, with even first iterations beating unprompted baselines before any adversarial feedback.

## Scaling to Multi-Agent Systems

The pattern scales to full-stack development through a three-agent architecture: planner expands simple prompts into detailed specs, generator works one feature per sprint, and evaluator exercises the running application via browser automation.

Before each sprint, generator and evaluator negotiate a **sprint contract** — agreeing on what "done" looked like before any code was written. Sprint 3 alone had 27 criteria covering the level editor. This creates verifiable success criteria rather than vague user stories.

The economics are stark: a solo agent built a game in 20 minutes for $9, but the full harness took 6 hours for $200 — "over 20x more expensive, but the difference in output quality was immediately apparent." The solo run's game was broken; the harness build actually worked.

## The Context Reset Innovation

Rajasekaran's work also advances our understanding of [context rot](/wiki/context-rot). While **compaction** summarizes earlier conversation to fit context limits, **context resets** start fresh agents with structured handoffs. Resets prevent "context anxiety" — models wrapping up prematurely as they approach perceived limits — which compaction can't fix because "it doesn't give the agent a clean slate."

This connects to broader patterns in [agent harness engineering](/wiki/agent-harness-engineering) where durable session state and clean handoff artifacts enable multi-session work that surpasses single-context capabilities.

## Infrastructure for Reliable Agents

Both developments point toward the same architectural shift: moving from prompt tweaks to systematic harness design. Whether evaluation methodology distributed as skills or adversarial training loops, the focus shifts to the substrate around the model — execution environment, feedback loops, and verification gates — as the primary driver of reliability.

This fits [Drew Breunig](/wiki/drew-breunig)'s economic observation that once frontier capability stopped arriving at flat prices, teams "started to think about what work went where." Harness and evaluation investment stopped being throwaway glue and became durable competitive advantages.

The convergence suggests a maturing field: instead of hoping better prompts will solve reliability, we're building the infrastructure that makes agent outputs verifiable, improvable, and trustworthy at scale.

---

## Loop and Graph Engineering: The Dual Topologies of Agentic Systems
- URL: https://pyweb.dev/writing/loop-and-graph-engineering
- Raw Markdown: https://pyweb.dev/writing/loop-and-graph-engineering.md
- Date: 2026-08-24

# Loop and Graph Engineering: The Dual Topologies of Agentic Systems

Modern AI engineering has moved past simple completion prompts into autonomous agent architectures. As practitioners scale these systems from prototype scripts to long-horizon enterprise engines, two foundational paradigms have crystallized: **Loop Engineering** and **Graph Engineering**.

Understanding when to run in a tight execution loop versus when to structure execution as an explicit state graph is the difference between an agent that reliably ships code and one that burns tokens into [context rot](/wiki/context-rot).

---

## 1. Loop Engineering: Tools in a Bounded Iteration

At its simplest, an LLM agent is defined as **an LLM running tools in a loop to achieve a goal** (as articulated by [Simon Willison](/wiki/simon-willison)). 

```mermaid
flowchart LR
    UO["User Objective"] --> LLM["LLM Decides Action"]
    LLM --> TE["Tool Execution"]
    TE --> TO["Tool Output"]
    TO --> LLM
```

In [Loop Engineering](/wiki/agentic-engineering-patterns), the core focus is local mechanical feedback:

- **Tight Tool Feedback:** The agent exercises its own code immediately via terminal test runners or linters ([TDD with agents](/wiki/tdd-with-agents)).
- **Brute Force Problem-Solving:** Given clear terminal error messages and deterministic constraints, the agent iterates until green without requiring human steering on every turn.
- **The Smart Zone Limit:** Loops excel within the model's peak reasoning window — the [smart zone](/wiki/smart-zone), typically the first ~150k tokens of a session on frontier models. Once execution transcripts accumulate dozens of intermediate bash runs and stack traces, the loop suffers from [prompt bloat](/wiki/prompt-bloat) and degrades.

---

## 2. Graph Engineering: Partitioned Topologies and State Separation

When a task exceeds a single context window or requires multi-disciplinary exploration (such as large codebase refactors or open-ended literature synthesis), running a single loop is a recipe for catastrophic context decay.

This is where **Graph Engineering** takes over. [Anthropic](/wiki/anthropic)'s production Research system describes this as "a multi-agent architecture with an orchestrator-worker pattern, where a lead agent coordinates the process while delegating to specialized subagents that operate in parallel" ([source](/wiki/raw/articles/anthropic-multi-agent-research-system-2025)):

```mermaid
flowchart LR
    LO["Lead Orchestrator"] --> SA["Subagent A: Web Explorer"]
    LO --> SB["Subagent B: Code Inspector"]
    LO --> SC["Subagent C: Eval Runner"]
    
    SA -->|Artifact / Summary| SYN["Synthesizer / Critic"]
    SB -->|Artifact / Summary| SYN
    SC -->|Artifact / Summary| SYN
```

- **Topological Separation of Concerns:** Rather than forcing one agent to hold the full state, work is delegated across dedicated worker nodes with isolated context windows ([multi-agent orchestration](/wiki/multi-agent-orchestration)).
- **Artifacts over Telephone Games:** Subagents do not stream raw tokens back to the coordinator. They write structured markdown artifacts or files to disk and return concise summaries ([handoff artifacts](/wiki/handoff-artifacts)).
- **Deterministic Gates:** Transition edges between graph nodes are guarded by programmatic assertions, schema validators, or automated fitness functions ([conformance suites as fitness functions](/wiki/conformance-suites-as-fitness-functions)).

---

## 3. The Synthesis: Composing Loops Inside Graph Nodes

Production AI architectures do not choose between loops and graphs—they nest them.

1. **The Graph defines the macro workflow:** High-level state transitions, specification drafting, ticket decomposition, and consensus gates.
2. **The Loop executes the micro task:** Inside an individual graph node (e.g., implementing a single ticket), a focused worker agent operates in a rapid, uninhibited tool loop until its verifiable test passes.

By isolating the execution loop to a clean subagent and returning only verified diffs and test results to the parent graph, you maintain zero context leakage while harnessing full agentic autonomy.

---

## Hello world
- URL: https://pyweb.dev/writing/hello-world
- Raw Markdown: https://pyweb.dev/writing/hello-world.md
- Date: 2026-08-22

This is the first post on pyweb.dev. It exists to prove the pipeline works: Markdown in, static HTML out, deployed to Firebase Hosting.

More will follow.

---

## Building Effective Agentic Workflows
- URL: https://pyweb.dev/writing/building-effective-agentic-workflows
- Raw Markdown: https://pyweb.dev/writing/building-effective-agentic-workflows.md
- Date: 2026-08-22

# Building Effective Agentic Workflows

The current plateau in AI-assisted software development comes from using models merely as inline autocomplete. When you hand an AI agent a complex repository with zero constraints, it writes code fast—and without careful boundaries, it rapidly accelerates codebase decay.

To build software reliably with autonomous agents, you need a disciplined engineering harness. Here are the core architectural patterns required to transform unpredictable LLM outputs into verifiable production artifacts.

---

## 1. Context Engineering: Curing Context Rot

Agents do not fail because they lack intelligence. They fail because they drown in noisy context. 

As an agent explores files, executes test suites, and reads logs, intermediate tokens accumulate. This triggers [context rot](/wiki/context-rot)—a sharp degradation in attention where the model forgets edge cases, hallucinates nonexistent APIs, and contradicts earlier constraints.

To maintain peak reasoning, keep the agent in its [smart zone](/wiki/smart-zone) (typically 0 to 150k tokens on frontier models):

- **Never run `/init`:** Avoid automated tooling that dumps hundreds of generic rules into your project config. Hand-craft a minimal [AGENTS.md specification](/wiki/agents-md-spec) at your workspace root containing only the high-signal facts: layout, build commands, and hard guardrails.
- **Kill [prompt bloat](/wiki/prompt-bloat):** Apply the defensibility test. If you cannot defend why an instruction is present with a concrete historical failure, cut it.
- **Enforce [progressive disclosure](/wiki/progressive-disclosure):** Expose lightweight indexes first. Let the agent load deep documentation or specialist tools on-demand using specific tool calls rather than flooding the system prompt up front.

---

## 2. Phase Boundaries and Handoff Artifacts

Long coding sessions that try to take an idea from fuzzy concept to fully deployed feature in a single context window will inevitably fail. 

Structure your workflow into discrete, bounded phases:

```mermaid
flowchart LR
    A["Grilling / Clarification"] --> B["Spec & Architecture"]
    B --> C["Ticket Decomposition"]
    C --> D["Fresh TDD Implementation"]
```

At each phase boundary, do not carry over the raw conversation transcript. Instead, generate a structured [handoff artifact](/wiki/handoff-artifacts):

1. **The Objective:** Exactly what problem is being solved.
2. **Decisions & Constraints:** Explicit architectural choices and discarded alternatives.
3. **The Frontier:** A list of unblocked, verifiable tasks.

When beginning implementation, clear the context (`/clear`) and seed the fresh session exclusively with the handoff artifact and the immediate ticket.

---

## 3. Tracer Bullets over Prototypes

When starting a new feature, avoid building wide horizontal slices (e.g., implementing five API stubs that return mock data). Instead, ship a [tracer bullet](/wiki/tracer-bullets).

A tracer bullet is a minimal, end-to-end slice that traverses every layer of your target architecture—from the database schema to the API endpoint and the UI component. 

Unlike throwaway prototypes, tracer bullets use production-ready error handling and testing seams. They establish the verified pipeline early so subsequent feature additions can build upon proven infrastructure.

---

## 4. Test-Driven Development as an Agent Seam

The single best harness for an autonomous coding agent is a failing test.

Never instruct an agent to "fix the bug" or "build the feature" in the abstract. Instead, follow strict [TDD with agents](/wiki/tdd-with-agents):

1. **Red Phase:** Direct the agent to write a minimal test reproducing the bug or asserting the new behavior. Execute the test and verify that it fails for the expected reason.
2. **Green Phase:** Write the minimal implementation code necessary to turn the test green.
3. **Refactor & Review:** Run automated linters and independent code review passes before merging.

Nature and production cannot be fooled. An agent that validates its work against deterministic terminal test outputs will consistently outperform one guided only by conversational feedback.

---