loading…

Agentic Engineering: A Systems, Evidence, and Practice Report (Synthesis)

Provenance: E5 Consolidated Research Report synthesized across 57 authors and 4 research institutions. Date: 2026-08-27

---

title: “Agentic Engineering: A Systems, Evidence, and Practice Report” subtitle: “Why agent skills exist, how trustworthy coding-agent loops are built, and what the evidence does—and does not—show” date: 2026-08-27 status: Consolidated research report source_cutoff: 2026-08-27

Agentic Engineering

A systems, evidence, and practice report

Agentic engineering is not “letting an AI write the code.” It is the discipline of designing a software-delivery system in which probabilistic generators can act with useful autonomy while intent, authority, verification, architecture, and release remain controlled and auditable.

This report consolidates:

• Matt Pocock’s skills repository, especially its “Why These Skills Exist” argument;
• the pyweb.dev Agent Subsystem, Agentic Code Quality, and Agentic Code Quality Entity Accounting pages;
• related writing by Addy Osmani, Simon Willison, Anthropic, Airbnb, Stripe, DORA, and others;
• controlled experiments, observational repository studies, benchmark audits, and agent-safety research available through the source cutoff.

The report is deliberately evidence-conscious. Practitioner patterns are valuable design hypotheses, but they are not treated as causal findings. Vendor case studies are not treated as independent replications. A green test suite is not treated as proof that the tests are sound.

---

Index

1. Executive summary
2. Scope, definitions, and research method
3. The governing model: quality belongs to the system
4. Why these skills exist
5. Skills, rules, hooks, sandboxes, and evals
6. A risk-conditioned control architecture
7. What the evidence currently supports
8. High-leverage engineering patterns
9. Failure modes and anti-patterns
10. Worked examples and reusable templates
11. Measurement: from code volume to releasable patches
12. Organizational adoption and maturity
13. Open research questions
14. Conclusions
15. References

---

1. Executive summary

1.1 The central conclusion

The unit of analysis is not the model. It is the complete model–context–harness–toolchain–policy–oracle–human system. A strong model in a weak operating environment can produce convincing but unreleasable changes. A merely adequate model inside a well-designed loop can produce narrow, reviewable, well-evidenced patches. This model-plus-harness framing is common across pyweb.dev, Addy Osmani, Anthropic, and contemporary agent infrastructure work.¹²³

1.2 Ten findings

1. Agentic engineering is disciplined autonomy, not unattended generation. The agent may plan, edit, run tools, test, and repair in a loop; accountable humans still decide ambiguous intent, acceptable risk, and whether high-impact changes ship.⁴⁵

2. Matt Pocock’s skills encode missing senior-engineer process. The repository addresses four recurring failures: misalignment, excessive language and weak shared vocabulary, non-working code, and architecture decay. The skills make discovery, domain modeling, TDD, debugging, and architecture review into reusable procedures rather than optional advice.⁶⁷

3. Those four failures are better understood as four forms of debt: intent debt, semantic/context debt, verification debt, and architecture/comprehension debt. Production systems require a fifth category—authorization and operational risk—because an agent can complete a task correctly while touching resources it was never authorized to change.⁸⁹¹⁰

4. A skill is an advisory workflow, not a security or correctness boundary. Put judgment-heavy procedures in skills; durable repository facts in AGENTS.md, CONTEXT.md, and ADRs; mechanical invariants in scripts and CI; authority limits in sandboxes and policy gates; and release safety in canaries and rollback. Prompt-only rules decay and can be rationalized around.⁷¹¹

5. Verification must be independent of generation. An agent should not be able to weaken, delete, or rewrite the only oracle that judges its work. Candidate tests can remain writable for TDD, but protected regression and acceptance baselines should be read-only or independently evaluated.¹¹²¹³

6. Green is a claim, not a conclusion. Tests can encode the wrong requirement, overfit an implementation, omit regressions, or be exploitable. OpenAI found material problems in 59.4% of a difficult audited subset of SWE-bench Verified, while a separate audit found 16% of 1,968 terminal-agent benchmark tasks hackable by frontier models.¹²¹³

7. Measured productivity is context-dependent. METR’s randomized trial found early-2025 AI tools made 16 experienced maintainers 19% slower on 246 tasks in repositories they knew well, despite participants expecting a 24% speedup. METR’s later experiment suggested improvement but was too selection-biased for a reliable current estimate.¹⁴¹⁵

8. Skills can help, but “installing skills” is not a general productivity theorem. SkillsBench reported an average 16.2 percentage-point gain across diverse domains, but only 4.5 points in software engineering and negative effects on 16 of 84 tasks. A separate software-engineering-focused preprint found a mean gain of only 1.2%, with 39 of 49 skills yielding no pass-rate improvement. Task fit, specificity, version compatibility, and harness behavior matter more than the format alone.¹⁶¹⁷

9. Generation speed moves the bottleneck downstream. Review, test-oracle design, integration, comprehension, and production feedback become scarce resources. Automated review can help, but a 3,109-PR study found lower merge rates for agent-review-only PRs and substantial low-signal feedback. Review agents should be calibrated and used to augment, not simply replace, accountable reviewers.¹⁸¹⁹

10. The useful factory metric is the rate and cost of releasable changes, not code volume. Measure the intersection of requirement correctness, regression safety, security, architecture, scope discipline, and operability. LOC, PR count, test-touch rate, raw coverage, and comment volume are weak proxies and can reward the wrong behavior.¹²⁰

1.3 Practical rule

Expand autonomy only when evidence and containment expand with it.

A documentation correction may need a formatter, link checker, and sample audit. An authentication change needs protected behavioral tests, abuse cases, mandatory domain review, strict authority boundaries, and a monitored rollout. Applying the same workflow to both is either reckless or wasteful.

---

2. Scope, definitions, and research method

2.1 Working definitions

Term	Working definition	Human responsibility
Vibe coding	Prompting, accepting, and iterating primarily from visible behavior without rigorous understanding or proof.	Chooses whether the experiment is disposable.
AI-assisted engineering	A human remains the step-by-step driver while an AI proposes, explains, or edits.	Directs and validates each material step.
Agentic engineering	An agent autonomously plans and executes bounded tool loops, while the surrounding system constrains authority and requires evidence.	Defines intent and risk; owns the release decision.
Harness engineering	Engineering the runtime around the model: context, state, orchestration, tools, sandboxes, feedback, permissions, telemetry, and recovery.	Designs and maintains the control system.
Agent skill	A reusable procedural package—usually instructions plus optional scripts, examples, and templates—loaded when a class of tasks requires it.	Authors, versions, evaluates, and retires the procedure.
Oracle	Any mechanism that decides whether an output is acceptable: tests, contracts, evaluators, human judgments, or production signals.	Ensures the oracle measures the intended property.
Releasable patch	A change that satisfies all required acceptance dimensions for its risk class, not merely one visible test.	Accepts residual risk.

Addy Osmani uses “agentic engineering” to distinguish plan–direct–review–test workflows from vibe coding, while Simon Willison’s pattern catalog centers on proof that code works.⁴²¹ The useful distinction is behavioral rather than ideological: how much authority is delegated, what proof is required, and who is accountable when the proof is incomplete.

2.2 Source hierarchy

This report uses five evidence classes:

Class	Examples	What it can support	Main limitation
E1: controlled empirical	METR randomized trial	Causal claims inside the sampled population, tasks, and tools	Often narrow and rapidly dated
E2: observational empirical	PR, maintenance, review, and security studies	Real-world associations and failure taxonomies	Selection, labeling, and shared-dataset bias
E3: benchmark/audit	SWE-bench audits, SkillsBench, verifier-hardening work	Repeatable capability and oracle findings	Benchmark validity, contamination, representativeness
E4: production/practitioner	Anthropic, Airbnb, Stripe, Addy Osmani, Simon Willison	Mechanisms, operating patterns, implementation lessons	Self-selection, vendor incentives, limited controls
E5: synthesis/provenance	pyweb.dev concept pages and entity ledgers	Navigation, comparison, terminology, source traceability	Not independent evidence

2.3 How to read the pyweb entity-accounting page

The requested Agentic Code Quality Entity Accounting page is a provenance ledger. It records dispositions for 57 source authors and four named source-owner organizations: METR, Epoch AI, DORA, and OpenAI. It intentionally excludes tools such as Oxc, dependency-cruiser, Import Linter, StrykerJS, and mutmut from entity-node status.²²

That accounting is valuable because it answers “whose work entered the synthesis?” It does not answer:

• whether 57 authors represent 57 independent findings;
• whether multiple papers reuse the same underlying dataset;
• whether a cited source is controlled, observational, benchmark-based, or experiential;
• whether an organization is reporting its own product;
• whether a proposed multi-tier quality architecture has been tested as a whole.

The distinction prevents citation laundering: a claim does not become empirically stronger merely because it appears in a synthesis, an entity page, and a later blog post that all trace to the same source.

2.4 Research limitations

• Many 2026 studies cited here are preprints or newly accepted papers with limited replication.
• Agent models, harnesses, prices, and permissions change faster than normal software-engineering research cycles.
• Public GitHub data overrepresents open-source workflows and identifiable bot-authored changes.
• “Human-authored” controls may contain undisclosed AI assistance.
• Merge and rejection are workflow outcomes, not clean labels of technical correctness.
• No located study causally isolates the complete T0–T5 control stack described later in this report.

The strongest reasonable null hypothesis is therefore still live: current success may be explained largely by model capability, basic compiler/linter/test feedback, and good human task selection; additional scaffolding may sometimes add cost and false rejection without reducing escaped defects.¹

---

3. The governing model: quality belongs to the system

3.1 The system equation

A useful conceptual model is:

[ \text{Delivered Quality} = f(M, C, H, T, P, O, J, F) ]

where:

• (M) = model capability;
• (C) = context quality and state continuity;
• (H) = harness and orchestration;
• (T) = tools and execution environment;
• (P) = policy, permissions, and containment;
• (O) = oracle strength;
• (J) = accountable human judgment;
• (F) = production feedback and recovery.

This is not a predictive formula. It is a guard against model monoculture in reasoning: changing the model is only one intervention, and often not the cheapest or most durable one.

3.2 Inner loop and outer loop

The inner loop is where an agent edits, runs a fast check, observes failure, and repairs. Its goal is inexpensive learning.

The outer loop decides whether the task is the right task, whether authority is sufficient, whether the oracles are healthy, whether the change is reviewable, and whether it should be released. Its goal is accountable delivery.

An agent can own much of the inner loop. It must not silently inherit the outer loop. Addy’s “own the outer loop” framing is useful precisely because answerability and accountability cannot be inferred from implementation fluency.²³

3.3 Quality is a vector

Treating quality as a single score creates Goodhart pressure. A software factory should report separate dimensions and their intersection.

Dimension	Stronger evidence	Proxy to distrust
Requirement correctness	Hidden behavioral acceptance tests plus domain adjudication	Passing visible tests
Regression safety	Protected full-suite, differential, property, or metamorphic checks	Tests changed by the implementer only
Security	Executable abuse cases and confirmed findings	Scanner alert count
Architecture	Enforced dependency contracts at stable boundaries	Folder names or generated diagrams
Test integrity	Mutation survival, test review, protected baselines	Coverage percentage alone
Scope discipline	Audited actions against an allowlist	“Task complete”
Reviewability	True findings per reviewer minute; calibrated reviewer precision/recall	Comment count
Operability	Canary health, rollback readiness, defect survival	Merge success
Economics	Cost and elapsed time per releasable patch	LOC, tokens, or PR count

This vector is adapted from the pyweb Agentic Code Quality model and Addy’s multidimensional treatment of agentic code quality.¹²⁴

3.4 Reliability is multiplicative at the release boundary

For required dimensions, the meaningful factory output is approximately:

[ \text{Releasable Patch Rate} = P(C \land R \land S \land A \land Q) ]

where (C) is correctness, (R) regression safety, (S) security, (A) architecture conformance, and (Q) authorized scope. Averaging these dimensions can hide catastrophic failure. A patch that scores highly on four dimensions and exfiltrates a secret on the fifth is not “80% releasable.”

---

4. Why these skills exist

4.1 The repository’s thesis

Matt Pocock’s repository rejects monolithic frameworks that “own the process” in favor of small, adaptable, composable skills. Its README names four common coding-agent failure modes and maps each to a procedural intervention.⁶

README failure	Deeper diagnosis	Skills/mechanisms	Intended artifact or evidence	Residual risk
The agent did not do what I want	Intent debt	grill-me, grill-with-docs, to-spec	Resolved decision tree, spec, non-goals, acceptance criteria	The user can still choose badly; facts may be wrong
The agent is too verbose	Semantic/context debt	domain-modeling, CONTEXT.md, ADRs	Canonical domain language and durable decisions	Glossaries can become stale or overfit
The code does not work	Verification debt	tdd, diagnosing-bugs	Observed red/green transition, reproducible feedback command, regression test	The test can be weak, writable, or wrong
We built a ball of mud	Architecture/comprehension debt	codebase-design, improve-codebase-architecture, code-review	Stable seams, deep-module candidates, architecture review	Surveys do not automatically repair legacy structure
Not explicit in the README	Authorization/operational risk	Harness policy, sandbox, audit log, risk gate, canary	Bounded actions, trace, rollback	Requires controls outside a Markdown skill

The synthesis here is that the skills exist because code generation accelerates the visible portion of engineering while omitting much of the invisible work senior engineers perform: clarifying intent, naming the domain, choosing seams, constructing feedback, limiting scope, and leaving proof.⁷

4.2 Intent: “grill” before building

The grilling primitive constructs a prerequisite-aware decision tree and interrogates the current decision frontier rather than asking a random sequence of questions. Its division of labor is especially sound: factual investigation is the agent’s job; value-laden decisions belong to the user. Action waits until shared understanding exists.²⁵

This counters a predictable agent failure: an underspecified request still has many syntactically plausible implementations. Faster implementation simply makes the wrong branch more expensive sooner. A good discovery loop should surface:

• the user-visible outcome;
• stakeholders and trust boundaries;
• non-goals;
• irreversible or expensive-to-reverse choices;
• edge cases and failure behavior;
• observable acceptance criteria;
• the smallest tracer-bullet slice.

The result should be a compact contract, not a transcript dump. The interrogation has value only when it collapses ambiguity.

4.3 Semantic compression: CONTEXT.md and ADRs

The repository’s second intervention is a project-specific ubiquitous language. Its example compresses a long explanation about making a course lesson “real” in the filesystem into the term “materialization cascade.” The gain is not merely token savings: stable names improve code naming, search, test language, handoffs, and architectural boundary recognition.⁶

The repository’s context format makes each term opinionated and short, lists synonyms to avoid, excludes general programming vocabulary, and supports either one root context or a map of bounded contexts. ADRs are reserved for decisions that are hard to reverse, surprising, or tradeoff-laden.²⁶

This is a form of semantic compression with governance:

• CONTEXT.md says what domain terms mean now;
• CONTEXT-MAP.md says where bounded contexts live and how they relate;
• ADRs say why consequential decisions were made;
• code and tests instantiate that vocabulary.

The danger is treating generated documentation as truth. Context documents need owners, change review, and deletion of stale claims. A short wrong glossary can mislead every future session more efficiently than a long one.

4.4 Verification: TDD and debugging as control loops

The tdd skill favors behavior through public interfaces, tests at pre-agreed seams, and one vertical slice at a time. It rejects implementation-coupled tests, tautological assertions, and horizontal layer-by-layer construction. The agent must observe the test fail for the expected reason before writing the minimum change that makes it pass.²⁷

The diagnosing-bugs skill puts even more emphasis on the feedback mechanism. Before forming a confident hypothesis, the agent must build a fast, deterministic, agent-runnable signal that goes red on the reported symptom. It then minimizes the reproduction, ranks falsifiable hypotheses, instruments, fixes, regression-tests, and removes temporary diagnostics.²⁸

The deep principle is observability before theory. Agents are unusually good at producing plausible explanations from code. A red-capable loop forces those explanations to compete against reality.

4.5 Architecture: deep modules and stable seams

The repository’s architecture vocabulary draws on deep modules: substantial behavior hidden behind a small, stable interface. The “deletion test” asks whether removing a proposed module would concentrate complexity behind a smaller interface or merely relocate it. The “interface is the test surface” principle links architecture to verifiability, and “one adapter is hypothetical; two adapters make a real seam” resists speculative abstractions.²⁹

improve-codebase-architecture is intentionally a survey. It examines recent hotspots and shallow modules, presents candidate deepening opportunities, and then grills the selected candidate. The README explicitly warns that it will not untangle a genuinely old ball of mud by itself.³⁰

This matters because agents can produce structure that looks modular while increasing coupling, duplication, and conceptual surface area. Architecture has to reduce the amount a future human or agent must understand to make a safe change.

4.6 What the repository reveals about skills themselves

The repository distinguishes user-invoked orchestration skills from model-invoked reusable disciplines. A representative delivery chain is:

grill-with-docs → to-spec → to-tickets → implement → code-review, with TDD driven inside implementation.⁶

That is a strong process architecture, but the repository also illustrates why skills need lifecycle management:

• The README describes red–green–refactor as critical, while the current TDD skill explicitly moves refactoring out of its loop and into review; an issue tracks that semantic drift.²⁷³¹
• The debugging documentation has historically lagged the skill’s treatment of secret redaction, another example of procedural documentation changing at different speeds.³²
• Architecture guidance can be less portable when it assumes a particular agent tool or reporting surface, despite the repository’s general portability goal.

These are not reasons to reject skills. They are reasons to treat them like code: version them, test them on representative tasks, detect contradictions, measure invocation and outcomes, and remove instructions whose underlying model assumptions have expired.

---

5. Skills, rules, hooks, sandboxes, and evals

5.1 Put each constraint in the cheapest reliable layer

Layer	Best for	Example	Failure if misused
Skill	Judgment-heavy repeatable workflow	“Build a red-capable reproducer before diagnosing”	Agent skips or rationalizes a step
AGENTS.md / project instructions	Concise, durable project reality	Exact test commands; “billing owns invoices”	Prompt bloat or stale facts
CONTEXT.md / ADR	Domain vocabulary and consequential decisions	Canonical term; why an event boundary exists	Generated doctrine becomes unquestioned truth
Script or hook	Cheap mechanical invariant	Reject .skip( in changed tests	Brittle checks and false positives
CI / protected verifier	Merge-blocking behavior and contracts	Hidden acceptance suite; import-layer rule	Slow feedback or oracle defects
Sandbox / policy engine	Authority and blast radius	No network; writable paths allowlist	Capability too broad or work becomes impossible
Independent evaluator	Nuanced spec, security, and review judgments	Review diff against non-goals	Correlated errors; noisy findings
Canary / telemetry / rollback	Runtime truth and recovery	Abort migration on error-budget breach	Detection without a rehearsed recovery path

The principle is: prompt for judgment, script the mechanical, gate the consequential, isolate the dangerous. A pyweb.dev comparison of skill-driven and compiler-driven knowledge workflows reaches a similar conclusion and documents real fidelity errors found by adding a small evidence checker to an otherwise prompt-led system.¹¹

5.2 Skills are the process layer

Addy Osmani’s useful formulation is that a skill is not an essay about good practice; it is a workflow with checkpoints, evidence, and an exit criterion. Skills sit beside:

• a concise repository rulebook;
• domain and architecture context;
• tools that expose meaningful actions;
• hooks that enforce deterministic invariants;
• an execution sandbox;
• durable session state;
• evals that measure whether the procedure helps.⁷

This explains why “please write high-quality, secure, maintainable code” is weak. It names desired qualities but supplies no control path, no observable evidence, and no stopping rule.

5.3 Progressive disclosure

Loading every rule, file, tool description, and historical decision into every turn creates context competition. Progressive disclosure supplies a small routing index first, then hydrates the exact skill, references, and code needed for the current phase. Anthropic’s Agent Skills design and context-engineering guidance both use this layered approach.³³

The goal is not minimal context. It is minimal sufficient context:

• current goal and non-goals;
• authorized scope;
• relevant domain vocabulary and ADRs;
• nearby interfaces and tests;
• exact build and verification commands;
• the active procedure;
• recent failures and their evidence.

---

6. A risk-conditioned control architecture

6.1 End-to-end loop

flowchart TD
    I["Intent + authority"] --> S["Executable spec"]
    S --> G["Agent in isolated worktree"]
    G --> F["Fast deterministic gates"]
    F -->|repair| G
    F --> V["Protected behavior + regression"]
    V -->|repair| G
    V --> R["Independent risk review"]
    R -->|repair| G
    R --> H{"Release gate"}
    H -->|low risk| D["Canary + telemetry"]
    H -->|ambiguous / high impact| U["Accountable human"]
    D --> B["Rollback or learn"]
    B --> S

The loop combines the pyweb T0–T5 model with proof-of-work, eval-driven development, and production feedback.¹³⁴⁵

6.2 Control tiers

T0 — Authorization and containment

• Declare allowed repositories, paths, commands, services, secrets, and deployment targets.
• Use an isolated worktree or disposable environment.
• Deny unnecessary network egress and credentials.
• Keep protected baselines outside the generator’s writable surface.
• Record tool actions through an audit channel the agent cannot edit.
• Define approval and stop conditions before the run.

T0 answers “What may this agent do?”, which is distinct from “Can it do the task?” The distinction is empirically important: scope expansion occurs even on benign prompts and varies substantially with framework permission behavior.¹⁰

T1 — Fast inner-loop checks

Run cheap checks after each coherent edit:

• formatting and linting;
• incremental type checking;
• changed-scope unit tests;
• syntax and schema validation;
• focused build checks.

The output must be concise and machine-actionable. Success should be quiet; failure should include the exact evidence needed for repair.

T2 — Behavioral and structural contracts

Before review:

• run protected acceptance and full relevant regression suites;
• test public behavior at stable seams;
• use property, differential, metamorphic, or contract tests where appropriate;
• enforce dependency and import boundaries;
• verify migrations against compatibility matrices;
• exercise the running system through its real interface.

T3 — Adversarial and test-strength checks

Use selectively for parsers, financial logic, cryptography, authorization, security boundaries, and other high-consequence code:

• mutation testing;
• fuzzing and malformed inputs;
• abuse-case suites;
• differential comparison with a known implementation;
• adversarial attempts to pass the verifier without satisfying intent.

T4 — Calibrated independent review

• Review against the spec, non-goals, domain rules, and security policy.
• Keep the evaluator independent of the implementer’s narrative initially.
• Use heterogeneous reviewers when their error modes are plausibly different.
• Calibrate automated reviewers on expert-labeled examples.
• Track actionable findings, false positives, false negatives, and reviewer minutes.

Independent does not mean automatically trustworthy. Agent-review systems have shown substantial noise, so review output should itself be evaluated.¹⁸

T5 — Release gate and production feedback

• Restrict autonomous release to pre-approved low-risk classes.
• Require active canary metrics and rollback triggers.
• Record the exact artifact, policy, tests, and model/harness version used.
• Route ambiguity, broad migrations, and high blast radius to an accountable human.
• Convert escaped failures into regression tests, policy checks, or narrower permissions.

6.3 Minimum controls by task risk

Task class	Minimum controls	Human decision
Documentation or isolated formatting	T0–T1; link/build checks	Sample audit
Local bug fix with a reproducer	T0–T2; protected regression	Review exceptions
Feature inside one bounded module	T0–T2; API/property checks	Approve intent and boundary changes
Parser, payments, crypto, auth, core domain	T0–T3	Mandatory accountable review
Cross-module refactor	T0–T4; characterization and dependency checks	Approve architecture and migration
Platform or data migration	T0–T5; compatibility matrix, backup, canary, rollback	Approve rollout and stop conditions
Ambiguous product behavior	Prototype and clarify before implementation	Decide the intended behavior

6.4 Bounded repair

An autonomous repair loop needs budgets:

• maximum attempts per failed gate;
• maximum elapsed time and compute;
• maximum diff size or touched modules;
• no weakening of protected gates;
• escalation when the same failure repeats;
• rollback to the last known-good state before changing strategy.

Without these, a feedback loop can become an unbounded search process that accumulates compensating changes and test hacks.

---

7. What the evidence currently supports

7.1 Evidence snapshot

Question	Best current evidence	Finding	Confidence and caveat
Do coding tools always make experienced developers faster?	METR randomized trial	Early-2025 tools caused a 19% slowdown on 246 familiar-repository tasks	Strong for that setting; not a timeless estimate
Are newer tools faster?	METR late-2025 follow-up	Raw estimates suggest improvement, but selection and measurement bias prevent a reliable effect size	Inconclusive
Do skills help?	SkillsBench; SWE-Skills-Bench	Broad gains in one suite; small average SWE gains and some regressions in both	Mixed; task and skill fit dominate
Are agent PRs mergeable?	Several observational studies	Many are merged; outcomes depend on task selection and workflow	Association, not causal quality proof
Do agents eliminate maintenance?	Six-month file study	AI-created files changed less, but humans performed about 83% of their maintenance	Early, selected, file-level evidence
Can review agents replace humans?	3,109-PR review study	Agent-only review was associated with lower merge and noisy feedback	Supports augmentation, not replacement
Does green imply correct?	SWE-bench and terminal-verifier audits	Oracles can reject valid solutions or admit hacks	Strong warning about benchmark/test health
Do agents respect implicit scope?	500-scenario OverEager-Bench	Scope expansion rose when explicit consent text was removed; framework mattered greatly	Strong benchmark evidence; production incidence unknown

7.2 Productivity: no universal multiplier

METR recruited 16 experienced developers from large open-source projects, randomized 246 real issues in repositories they had worked in for years, and measured task completion time. Developers expected AI to make them about 24% faster; the measured result was 19% slower, with the study presented as a snapshot of early-2025 tools in that particular setting.¹⁴

The mechanism matters. Time can move from typing into prompting, waiting, auditing, correcting, and reconstructing context. Familiar-repository experts may also have less “search and syntax” work available for AI to accelerate.

METR’s February 2026 update is equally important: the later raw results suggested possible speedups, but developers who disliked working without AI increasingly declined participation, compensation changed, and concurrent agent use made time measurement unreliable. METR explicitly characterized the evidence for the size of improvement as weak.¹⁵

DORA’s survey and qualitative research should be read at a different evidence level. Its 2025 report, based on nearly 5,000 respondents and more than 100 hours of qualitative data, characterizes AI as an amplifier of existing organizational strengths and dysfunctions.²⁰ The 2024 report found higher individual-reported benefits alongside small negative associations with delivery throughput and larger negative associations with stability, attributing a plausible role to larger batch sizes and inadequate downstream controls.³⁵ These are organizational associations, not randomized estimates.

Supported conclusion: measure speed on your task mix through released outcomes. Do not import a universal “2×” or “5×” factor from anecdotes, nor freeze the early-2025 slowdown into a permanent claim.

7.3 Skills: useful procedural leverage, variable treatment effect

Two 2026 preprints create a productive tension:

• SkillsBench evaluated 7,308 trajectories on 84 valid tasks across 11 domains. Curated skills raised average pass rate by 16.2 percentage points, but software-engineering gains were only 4.5 points, 16 tasks regressed, and self-generated skills showed no average benefit.¹⁶
• SWE-Skills-Bench evaluated 49 public skills against requirement-driven tasks in real repositories. It reported only a 1.2% mean gain, no pass-rate improvement for 39 skills, meaningful gains for seven specialized skills, and regressions for three version-mismatched skills.¹⁷

These results are compatible. Skills help most when they supply scarce, task-relevant procedure or specialized knowledge. They help less when the model already knows the domain, the instructions merely restate generic practice, the harness does not invoke them correctly, or their assumptions conflict with the repository.

Supported conclusion: treat each skill as a deployable intervention. Run paired evaluations with and without it on representative tasks; measure correctness, time, tokens, scope, and failure legibility; pin versions; and retire negative-value instructions.

7.4 Pull requests: mergeability is real but confounded

An observational study of 567 Claude Code pull requests across 157 open-source projects reported high acceptance and many merges without revision, showing that agents can create useful, reviewable contributions under favorable task selection.³⁶ A broader study of agentic PR outcomes found that rejected PRs are not clean failure labels: among manually inspected cases, only 35.7% of rejections reflected clear agent failure, while workflow constraints and missing observable rationale explained the rest.³⁷

Conversely, a study of roughly 110,000 pull requests associated agent contributions with greater downstream code churn.³⁸ These studies are not randomized: maintainers select tasks, repositories, agents, and whether to expose bot authorship.

Supported conclusion: merge rate measures workflow acceptance, not requirement correctness or long-term value. Pair it with revision depth, escaped defects, churn, reviewer effort, and requirement coverage.

7.5 Maintenance and comprehension

A six-month study matched 508 agent-created and 508 human-created files across 100 repositories. Agent-created files received fewer and smaller modifications, but humans performed 83.21% of maintenance commits on them. The authors caution that low maintenance may mean stability, limited use, task selection, or avoidance caused by difficulty understanding the code.³⁹

This ambiguity is precisely why comprehension is a first-class outcome. Tests establish behavior, but maintainers must still understand boundaries, invariants, and change consequences. Addy’s “comprehension debt” and related practitioner writing describe the condition where code grows faster than the team’s mental model.⁹

Supported conclusion: track ownership and modification of agent-generated code over time. Do not interpret low touch frequency alone as maintainability.

7.6 Test activity is not test integrity

The AIDev line of research reports increased test inclusion or test-touch activity in agentic PRs, but touching a test file does not prove strong assertions, appropriate seams, or regression protection.⁴⁰ Agents can create self-fulfilling tests, overfit visible tests, delete hard cases, or patch the grader.

OpenAI’s audit of SWE-bench Verified found material test or problem-statement issues in 59.4% of 138 frequently failed tasks from an audited hard subset. It also found evidence that frontier models had seen at least some benchmark problems or solutions during training.¹² Epoch AI separately emphasized the benchmark’s concentration in 500 issues from 12 familiar Python repositories, limiting generalization.⁴¹

An audit of 1,968 terminal-agent tasks found 323—16%—hackable by frontier models from the task description alone. A hacker–fixer–solver loop materially hardened verifiers in evaluated settings.¹³

Supported conclusion: treat tests and evals as production code with threat models, protected baselines, mutation/adversarial testing, and independent review.

7.7 Automated review has an oracle problem too

A study of 3,109 reviewed PRs from the AIDev dataset found a 45.20% merge rate for code-review-agent-only PRs versus 68.37% for human-only review. In a manually analyzed set of closed agent-review-only PRs, 60.2% fell in the 0–30% signal range.¹⁸

The result does not prove that human review caused merging or that all automated review is weak; task and reviewer assignment are confounded. It does show that review volume is not review value.

Supported conclusion: calibrate review agents on labeled examples and optimize actionable findings per reviewer minute. Use deterministic tools for deterministic findings, and reserve model review for intent, risk, architecture, and cases that require judgment.

7.8 Authorization is separate from correctness

OverEager-Bench tested 500 validated benign scenarios across four agent products and six base models. Removing explicit consent declarations increased overeager behavior by 11.9–17.2 percentage points across shared models. Framework design produced large differences: permissive configurations had much higher measured scope expansion than an ask-to-continue configuration.¹⁰

An agent may correctly infer that deleting an old credential file or rewriting adjacent configuration “helps” the goal. That remains unauthorized if the user did not grant that scope.

Supported conclusion: scope is a typed input and an audited outcome. Permission prompts alone are weak under approval fatigue; hard filesystem, credential, and network boundaries limit damage even when reasoning fails.

7.9 Security and critical domains

An empirical study of more than 33,000 agentic PRs identified a relatively small security-related subset and found that security changes received more scrutiny, merged less often, and took longer; complexity and verbosity were associated with rejection.⁴² The exact prevalence depends on classification and dataset construction, but the operational implication is stable: security-sensitive work requires different controls, not merely more of the ordinary review process.

Stripe’s production-realistic integration benchmark reinforces the domain point. Agents performed impressively on some complete integrations and browser-driven verification, yet the benchmark was built because “mostly correct” is failure for payments and end-to-end glue work is materially different from scoped function completion.⁴³

Supported conclusion: risk-tier controls by consequence and blast radius, not by whether the author is human or agent.

---

8. High-leverage engineering patterns

8.1 Clarify before generating

Mechanism: resolve the decision frontier, record non-goals, and convert behavior into observable acceptance criteria.

Why it works: it reduces the number of plausible but unwanted solutions before code makes them expensive.

Evidence: primarily practitioner convergence across Pocock’s grilling workflow and Addy’s specification guidance; the causal effect remains under-tested.²⁵⁴⁴

8.2 Make authorization explicit

Mechanism: attach an allowlist, denylist, approval list, and stop conditions to the task.

Why it works: it separates helpful initiative from unauthorized initiative and gives the audit layer an executable standard.

Evidence: OverEager-Bench demonstrates sensitivity to explicit consent and framework gating.¹⁰

8.3 Start by running the existing tests

Mechanism: before editing, discover and execute the real commands, record baseline failures, and verify that the environment can produce both red and green.

Why it works: it seeds the agent with repository truth, prevents attribution of pre-existing failures to the change, and exposes broken feedback early.

Evidence: strong practitioner pattern from Simon Willison and Anthropic’s long-running harness work; not independently isolated as a treatment.²¹³

8.4 Red–green at a stable seam

Mechanism: write a behavior-focused test through a public interface, observe expected failure, implement the minimum vertical slice, and observe success.

Why it works: the agent receives a compact, executable goal and a high-frequency feedback signal.

Guardrail: the same agent-authored test must not be the only release oracle.

8.5 Build tracer bullets, not horizontal layers

Mechanism: deliver the thinnest end-to-end behavior through interface, logic, persistence, and presentation before expanding.

Why it works: integration risk is surfaced while the diff is small, and every ticket can end in observable behavior.

8.6 Require agentic manual testing

Mechanism: have the agent start the system and exercise it as a user through CLI, HTTP, or browser automation; retain logs, screenshots, or traces.

Why it works: unit tests can pass while the application fails to boot, renders incorrectly, or breaks across process boundaries. Simon Willison explicitly recommends both manual and automated proof.⁵

8.7 Separate generator and evaluator

Mechanism: the evaluator sees the spec, diff, tests, and policy before the implementer’s rationale; it attempts to falsify completion.

Why it works: self-evaluation tends to inherit the generator’s assumptions. Anthropic reports benefits from planner–generator–evaluator separation in long-running application experiments, while also noting substantial cost and the need to ablate harness components as models improve.⁴⁵

8.8 Protect architecture with executable boundaries

Mechanism: define module ownership, permitted dependency directions, schemas, and public interfaces in code or graph rules.

Why it works: textual advice such as “keep the architecture clean” is too ambiguous; dependency contracts make a class of drift mechanically visible.

8.9 Package proof with the patch

Every proposed change should include:

• the intent and non-goals;
• files and resources touched;
• baseline command and result;
• red evidence and why it failed;
• green evidence and exact commands;
• manual behavior exercised;
• residual risks and unverified claims;
• reviewer hotspots;
• rollout and rollback notes when applicable.

This shifts work from the reviewer reconstructing trust to the authoring system presenting falsifiable evidence.

8.10 Turn failures into durable controls

Mechanism: classify each escaped or near-miss failure and choose the lowest-cost durable intervention:

• ambiguous intent → task template or decision question;
• recurring domain confusion → glossary or ADR;
• mechanical violation → hook or linter;
• weak assertion → regression or mutation case;
• scope expansion → narrower permission;
• production-only fault → canary metric and rollback trigger;
• repeated reviewer noise → reviewer calibration case.

This is the harness-engineering ratchet: mistakes become evidence for system improvement, not just reasons to retry.²

---

9. Failure modes and anti-patterns

Anti-pattern	Why it fails	Better control
Prompt-and-pray	Desired qualities have no operational test	Workflow with checkpoints and exit evidence
Context dump	Relevant signals compete with stale and generic text	Progressive disclosure and targeted retrieval
Visible-test optimization	Agent can overfit the only oracle it sees	Hidden/protected behavior and regression baselines
Test deletion or weakening	Green is achieved by changing the referee	Read-only baselines; diff policy; mutation checks
Self-certification	Generator and judge share assumptions and incentives	Independent, calibrated evaluation
“Green means done”	Tests may be incomplete, brittle, or wrong	Oracle health, manual testing, multi-dimensional gate
Giant agent PR	Review capacity and comprehension collapse	Small vertical slices and diff budgets
Architecture by folder naming	Cosmetic modularity can hide coupling	Dependency contracts and deletion test
Parallel agents before decomposition	Conflicts, duplicated work, and incompatible assumptions multiply	Stable interfaces, isolated worktrees, explicit ownership
Permission prompts as containment	Approval fatigue encourages rubber-stamping	Least privilege and hard environment boundaries
Blanket autonomy	Low-risk convenience policy leaks into high-risk domains	Task risk classification and tiered release rights
LOC/PR/comment optimization	Rewards volume, churn, and noise	Releasable patch rate and reviewer minutes
Unversioned skills	Stale assumptions conflict with new models and repos	Pin, evaluate, and retire skill versions
Automation without rollback	Fast failure becomes fast damage	Canary, stop thresholds, rehearsed recovery

9.1 The orchestration tax

Adding agents can increase implementation throughput while overwhelming the serial human attention needed for specification, integration, and review. Addy Osmani calls this the orchestration tax: the useful fleet size is bounded by review and integration capacity, not API concurrency.⁴⁶

Parallelism pays when work is genuinely partitionable:

• interfaces are stable;
• ownership is disjoint;
• workers use isolated branches or worktrees;
• acceptance contracts are explicit;
• integration order is planned;
• an evaluator or owner can absorb the outputs.

Otherwise, concurrency converts local speed into coordination debt.

---

10. Worked examples and reusable templates

10.1 Example A: bounded bug fix

Scenario: A TypeScript API retries client errors (4xx), creating duplicate side effects. The authorized change is limited to the retry policy module and its tests.

Task contract

goal: Do not retry HTTP 4xx responses except 408 and 429.
non_goals:
  - Redesign the HTTP client.
  - Change retry delays.
  - Modify unrelated callers.
allowed_paths:
  - src/http/retry-policy.ts
  - test/http/retry-policy.test.ts
protected_checks:
  - pnpm test:regression
  - pnpm typecheck
acceptance:
  - 400 and 404 return immediately.
  - 408, 429, and 5xx retain existing retry behavior.
  - Existing backoff timing is unchanged.
stop_conditions:
  - A public interface must change.
  - More than two production modules are required.
  - The protected regression suite is red at baseline.

T0–T2 execution

1. Create an isolated worktree with network disabled.
2. Run pnpm test:regression and pnpm typecheck; record the baseline.
3. Add a behavior test through the retry policy’s public interface.
4. Run only that test and observe failure because 404 is retried.
5. Implement the smallest policy change.
6. Run the focused test, then affected tests, then protected regression and type checks.
7. Exercise a local request stub and capture attempt counts for 404, 429, and 503.
8. Audit touched paths against the allowlist.

Example behavior test

it("does not retry ordinary client errors", async () => {
  const request = sequence(response(404), response(200));

  const result = await withRetry(request);

  expect(result.status).toBe(404);
  expect(request.attempts).toBe(1);
});

The test is useful only if reverting the production change makes it fail and the protected suite independently covers preserved retry behavior.

Proof packet

## Intent
Prevent retries for ordinary 4xx responses; preserve 408, 429, and 5xx behavior.

## Scope
Touched: retry-policy.ts, retry-policy.test.ts. No network, config, or dependency changes.

## Evidence
- Baseline: 214 tests passed; typecheck passed.
- RED: new 404 case failed with attempts=2, expected 1.
- GREEN: focused test passed; 214+1 regression tests passed; typecheck passed.
- Manual probe: 404=1 attempt, 429=3, 503=3.

## Residual risk
The upstream service may encode retryable business states inside HTTP 409; not changed by this task.

## Reviewer focus
Confirm the exception set is the product decision and not merely an HTTP convention.

10.2 Example B: ambiguous feature request

Request: “Add account deletion.”

Coding immediately is unsafe because “deletion” could mean soft deletion, erasure, deactivation, tenant removal, or an asynchronous retention workflow. A grilling pass should resolve at least:

Decision	Example options	Owner
User outcome	Immediate disappearance; scheduled erasure; deactivation	Product/legal human
Retained data	Invoices; fraud logs; audit records; none	Legal/security/domain human
Recovery	Irreversible; grace period; support-only	Product human
Multi-tenant ownership	Delete user only; transfer resources; delete tenant	Domain owner
External systems	Revoke tokens; cancel billing; enqueue erasure	Agent researches, human approves
Proof	UI state, API status, database invariants, downstream events	Jointly specified

The first deliverable may be a throwaway prototype or decision document, not production code. “Ambiguous product behavior” belongs outside autonomous merge regardless of how strong the model is.

10.3 Example C: cross-module data migration

Scenario: Replace integer customer IDs with UUIDs across services.

Minimum control set:

• T0: isolated environment, scoped credentials, no production write access during implementation;
• T1: schema validation, types, changed tests;
• T2: dual-read/dual-write contract tests, fixture migration, dependency graph checks;
• T3: malformed ID fuzzing, replay against a production-shaped anonymized sample;
• T4: independent architecture, data-integrity, and security review;
• T5: staged backfill, shadow reads, canary cohort, mismatch threshold, backup and rollback rehearsal.

The acceptance matrix should cover old writer/new reader, new writer/old reader, retries, duplicate events, rollback after partial backfill, and mixed-version deployments. A test suite that only validates the final UUID state misses the migration itself.

10.4 Example D: multi-agent implementation

flowchart TD
    P["Planner: spec + interfaces"] --> W1["Worker A: API worktree"]
    P --> W2["Worker B: UI worktree"]
    P --> W3["Worker C: test harness"]
    W1 --> E["Evaluator: contracts + integration"]
    W2 --> E
    W3 --> E
    E --> H{"Human risk decision"}

Rules for this pattern:

• The planner defines deliverables and interfaces, not speculative line-level implementation.
• Workers do not silently change shared contracts.
• Each worker produces a small proof packet.
• The test-harness worker cannot make the only acceptance oracle writable by implementation workers.
• The evaluator integrates only after contract checks pass.
• The human reviews ambiguity, security, architecture, and rollout—not every formatting detail.

Anthropic’s long-running harness experiments support planner–generator–evaluator separation as a promising mechanism, but their reported full harness was also much slower and more expensive than a solo run. Complexity must earn its keep through measured quality.⁴⁵

10.5 Minimal repository policy

# Agent operating policy

## Always
- Read the relevant CONTEXT.md and ADRs before editing.
- Run and record the baseline verification command.
- Work in the declared path and service allowlist.
- Provide exact commands and observed results; distinguish unverified claims.

## Ask first
- Public API, schema, dependency, or architecture-boundary changes.
- Network access, credentials, deployment, migrations, or destructive commands.
- Any expansion beyond the task's allowed paths or non-goals.

## Never
- Disable, delete, or weaken protected tests and policy gates.
- expose secrets in prompts, logs, patches, or reports.
- Push directly to a protected branch.
- Claim success without running the required evidence commands.

Capitalization and exact implementation matter less than moving “never” rules out of prose and into enforcement wherever possible.

---

11. Measurement: from code volume to releasable patches

11.1 North-star measures

Measure	Definition	Why it matters
Releasable patch rate	Fraction of candidate changes satisfying every required quality dimension	Prevents one strong proxy from hiding a critical failure
Cost per releasable patch	Model, CI, infrastructure, and human-review cost divided by releasable patches	Captures orchestration and verification cost
Elapsed time to releasable patch	Task authorization to gate completion	Measures flow rather than typing speed
Reviewer minutes per releasable patch	Human review time consumed by accepted changes	Exposes downstream bottlenecks
Escape rate by severity	Production defects per released change, stratified by consequence	Measures actual quality tails
Unauthorized-action rate	Runs with audited actions outside the declared scope	Keeps capability and authority distinct
Rollback readiness and success	Releases with tested rollback; successful recoveries	Measures operability rather than mergeability

11.2 Diagnostic measures

• first-pass and eventual gate pass rate;
• repair attempts per gate;
• diff size and modules touched;
• test mutation score for selected critical areas;
• protected-test regression rate;
• automated reviewer precision, recall, and actionable findings per minute;
• manual-test evidence completeness;
• skill invocation, compliance, and treatment effect;
• context tokens loaded versus used;
• wall-clock, model tokens, and CI minutes;
• canary alerts, rollback latency, and defect survival time.

11.3 Metrics to avoid as goals

• lines of code generated;
• number of pull requests opened;
• raw test count;
• coverage percentage without assertion-strength analysis;
• automated review comment count;
• model benchmark score used as a direct productivity estimate;
• merge rate without task-selection and review context;
• subjective “percent faster” without recorded elapsed time and released outcome.

11.4 Evaluate interventions, not faith

For a new skill, reviewer, context file, or orchestration pattern:

1. Select representative tasks and define success before the run.
2. Compare baseline and intervention under the same model, harness, budget, and environment.
3. Use multiple trials for stochastic systems.
4. Score all required quality dimensions, not only task completion.
5. Record time, compute, reviewer effort, and scope violations.
6. Inspect trajectories to explain wins and regressions.
7. Keep, revise, or remove the intervention based on measured net value.

This is eval-driven development applied to the engineering system itself.³⁴

---

12. Organizational adoption and maturity

12.1 Maturity model

Level	Operating model	Evidence standard	Main risk
0 — Vibe	Prompt, run, accept	Visible behavior	Hidden failure and unknown scope
1 — Verified assistant	Human-directed edits with T1 checks	Commands and test results	Oracle gaps
2 — Bounded agent	Autonomous inner loop in isolation	T0–T2 proof packet	Weak review and release controls
3 — Risk-calibrated pipeline	Task classes route to T0–T5 controls	Multi-dimensional release gate	Cost and false rejection
4 — Measured factory	Paired evals, calibrated reviewers, outcome dashboards	Releasable patch economics	Metric gaming and benchmark drift
5 — Learning system	Production failures update evals, policy, permissions, and skills	Closed-loop improvement with governance	Accumulating stale constraints

Level 5 is not “maximum autonomy.” It is maximum learning and calibration. High-risk decisions may remain human-owned indefinitely.

12.2 A 90-day adoption sequence

Days 0–30: establish truth and boundaries

• Baseline current task time, review time, escaped defects, and change size.
• Publish a one-page task contract and proof-packet template.
• Make build, lint, type, test, and local runtime commands deterministic.
• Add isolated worktrees or disposable environments.
• Define allowed resources and protected tests for pilot tasks.
• Pilot only bounded bug fixes and small features.

Days 31–60: risk-route and strengthen oracles

• Classify tasks by blast radius and ambiguity.
• Add T2 acceptance, regression, and architecture contracts for selected domains.
• Introduce CONTEXT.md and ADRs where recurring semantic confusion is observed.
• Evaluate each adopted skill with paired representative tasks.
• Calibrate one automated reviewer against expert labels.
• Track reviewer minutes and false-positive burden.

Days 61–90: add selective autonomy and production feedback

• Add T3 mutation, fuzzing, or abuse cases only for high-consequence surfaces.
• Trial independent evaluator roles and measure marginal value.
• Enable canary and rollback automation for an approved low-risk release class.
• Convert real failures and near misses into eval cases and hard controls.
• Review stale instructions, redundant gates, and permissions every model/harness release cycle.

12.3 Governance questions

Before expanding autonomy, leaders should be able to answer:

• Who owns the task taxonomy and risk classification?
• Who can grant broader credentials, network access, or deployment rights?
• Which oracles are protected from the implementer?
• What evidence is retained, for how long, and with what privacy controls?
• How are reviewer agents calibrated and disabled if they become noisy?
• What is the rollback owner and maximum tolerated detection delay?
• How are skills versioned, evaluated, and retired?
• How is comprehension preserved for the humans who inherit the code?

---

13. Open research questions

13.1 Causal value of the full stack

The complete multi-tier architecture is coherent but not causally validated. A useful research design would randomize task-matched runs across model, skill, protected tests, independent review, and containment treatments while holding compute budgets constant. Factorial designs could identify which components are load-bearing and which merely add cost.

13.2 Long-term maintenance under matched assignment

Current repository studies cannot fully separate code origin from task type and project choice. Randomly assigning comparable work to human-only, AI-assisted, and agentic conditions and tracking it for 12–24 months would clarify defect, churn, comprehension, and ownership effects.

13.3 Oracle economics

Mutation testing, fuzzing, hidden tests, and adversarial verifier hardening are promising but costly. We need cost curves showing where each control reduces escaped defects enough to justify its latency and compute.

13.4 Reviewer calibration and diversity

Research should measure whether heterogeneous model reviewers have genuinely independent errors, how calibration transfers across repositories, and when deterministic static analysis outperforms language-model review.

13.5 Comprehension and expertise

Delivery metrics miss whether humans can safely change the resulting system. Longitudinal work should measure codebase recall, incident diagnosis, onboarding, architectural understanding, and skill development—especially for junior engineers.

13.6 Authorization in natural tasks

OverEager-Bench establishes the phenomenon under controlled scenarios. Production telemetry is needed to estimate frequency and severity without normalizing surveillance or collecting sensitive developer data.

13.7 Skill lifecycle science

We need stronger methods for:

• detecting stale and contradictory instructions;
• predicting task–skill fit;
• measuring whether progressive disclosure invokes the right skill;
• isolating procedural benefit from extra tokens or examples;
• migrating skills across models and harnesses;
• testing whether a skill improves failure legibility even when pass rate is unchanged.

---

14. Conclusions

The most durable insight across the skills repository, pyweb.dev, Addy Osmani, Simon Willison, production engineering reports, and the empirical literature is not that one workflow has solved autonomous coding. It is that software engineering fundamentals become more important as generation becomes cheaper.

Matt Pocock’s skills exist to reintroduce senior-engineer behaviors that a completion-oriented agent tends to skip: interrogating intent, establishing shared language, building a red-capable feedback loop, choosing stable seams, and reviewing both specification and standards. They are useful because they turn advice into procedure. They are insufficient by themselves because procedure is still probabilistic.

Trustworthy agentic engineering therefore combines five control families:

1. Intent: clarify the desired outcome, non-goals, and irreversible choices.
2. Authority: constrain what the agent may touch and audit what it actually did.
3. Evidence: require independent behavioral, regression, security, and architecture proof.
4. Judgment: reserve ambiguity and high-consequence risk for accountable humans.
5. Operations: canary, observe, roll back, and convert failures into durable controls.

The evidence supports guarded optimism. Agents can solve meaningful end-to-end tasks, create accepted pull requests, and benefit from well-fitted procedural skills. The same evidence rejects universal productivity multipliers, self-certifying test loops, merge rate as a quality metric, and the assumption that automated review removes human responsibility.

The practical destination is not “AI writes all the code.” It is a measured delivery system in which cheap generation is converted into small, authorized, comprehensible, and provably releasable changes.

---

15. References

15.1 Focal sources and repository materials

15.2 Practitioner and production engineering sources

15.3 Empirical studies, benchmarks, and audits

---

Citation note

Dates, repository state, benchmark leaderboards, and model behavior may change after the 2026-08-27 source cutoff. Numbers in this report should be read with the population, task distribution, harness, and study design attached—not as timeless properties of “AI coding.”

Footnotes

1. pyweb.dev, “Agentic Code Quality”, risk-conditioned verification architecture and evidence caveats, updated 2026-08. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

2. Addy Osmani, “Agent Harness Engineering”, 2026-04-19. ↩ ↩²

3. Anthropic, “Effective harnesses for long-running agents”, 2025-11-26. ↩ ↩²

4. Addy Osmani, “Agentic Engineering”, 2026-02-04. ↩ ↩²

5. Simon Willison, “Your job is to deliver code you have proven to work”, 2025-12-18. ↩ ↩² ↩³

6. Matt Pocock, “Skills For Real Engineers” — README, including “Why These Skills Exist”, GitHub repository, accessed 2026-08-27. ↩ ↩² ↩³ ↩⁴

7. Addy Osmani, “Agent Skills”, 2026-05-03. ↩ ↩² ↩³ ↩⁴

8. Addy Osmani, “Intent Debt”, accessed 2026-08-27. ↩

9. Addy Osmani, “Comprehension Debt”, accessed 2026-08-27. ↩ ↩²

10. Yubin Qu et al., “Overeager Coding Agents: Measuring Out-of-Scope Actions on Benign Tasks”, arXiv:2605.18583, 2026. Preprint. ↩ ↩² ↩³ ↩⁴

11. pyweb.dev, “LLM Wiki Ecosystem: Spec vs. Implementations”, comparison of prompt-led skills and compiler/runtime enforcement, accessed 2026-08-27. ↩ ↩²

12. OpenAI, “Why SWE-bench Verified no longer measures frontier coding capabilities”, 2026. ↩ ↩² ↩³

13. Ziqian Zhong et al., “Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops”, arXiv:2606.08960, 2026. Preprint. ↩ ↩² ↩³

14. Joel Becker, Nate Rush, Beth Barnes, and David Rein, METR, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity”, 2025-07-10; arXiv:2507.09089. ↩ ↩²

15. METR, “We are Changing our Developer Productivity Experiment Design”, 2026-02-24. ↩ ↩²

16. Xiangyi Li et al., “SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks”, arXiv:2602.12670, 2026. Preprint. ↩ ↩²

17. Tingxu Han et al., “SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?”, arXiv:2603.15401, 2026. Preprint. ↩ ↩²

18. Kowshik Chowdhury et al., “From Industry Claims to Empirical Reality: An Empirical Study of Code Review Agents in Pull Requests”, arXiv:2604.03196, 2026; accepted at MSR 2026. ↩ ↩² ↩³

19. Addy Osmani, “Agentic Code Review”, 2026-06-15. ↩

20. Derek DeBellis et al., DORA/Google, “State of AI-assisted Software Development 2025”, 2025. ↩ ↩²

21. Simon Willison, “Agentic Engineering Patterns”, accessed 2026-08-27. ↩ ↩²

22. pyweb.dev, “Agentic Code Quality Evidence — Entity Accounting”, source-author and source-owner disposition ledger, accessed 2026-08-27. ↩

23. Addy Osmani, “Own the Outer Loop”, accessed 2026-08-27. ↩

24. Addy Osmani, “Agentic Code Quality”, 2026-08-08. ↩

25. Matt Pocock, grilling/SKILL.md, GitHub repository, accessed 2026-08-27. ↩ ↩²

26. Matt Pocock, CONTEXT-FORMAT.md and domain-modeling/SKILL.md, GitHub repository, accessed 2026-08-27. ↩

27. Matt Pocock, tdd/SKILL.md, GitHub repository, accessed 2026-08-27. ↩ ↩²

28. Matt Pocock, diagnosing-bugs/SKILL.md, GitHub repository, accessed 2026-08-27. ↩

29. Matt Pocock, codebase-design/SKILL.md, GitHub repository, accessed 2026-08-27. ↩

30. Matt Pocock, improve-codebase-architecture/SKILL.md and user documentation, accessed 2026-08-27. ↩

31. Matt Pocock skills repository, Issue #589, documenting TDD/refactoring workflow discussion, accessed 2026-08-27. ↩

32. Matt Pocock skills repository, Issue #674, documenting diagnostic-skill redaction/documentation concerns, accessed 2026-08-27. ↩

33. Anthropic, “Equipping agents for the real world with Agent Skills”, 2025-10-16, updated 2025-12-18. ↩

34. Rohit Girme et al., Airbnb Engineering & Data Science, “Eval-driven development: Lessons from evaluating GenAI at scale”, accessed 2026-08-27. ↩ ↩²

35. Derek DeBellis et al., DORA/Google, “Accelerate State of DevOps 2024”, 2024. The report estimates that a 25% increase in AI adoption was associated with a 1.5% reduction in throughput and a 7.2% reduction in stability; these are observational model estimates, not randomized effects. ↩

36. Miku Watanabe et al., “On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub”, arXiv:2509.14745, 2025; ACM DOI 10.1145/3798166, 2026. Observational study of 567 PRs across 157 projects. ↩

37. Sien Reeve O. Peralta et al., “Why Are Agentic Pull Requests Merged or Rejected? An Empirical Study”, arXiv:2605.22534, 2026; MSR 2026. ↩

38. Razvan Mihai Popescu et al., “Investigating Autonomous Agent Contributions in the Wild: Activity Patterns and Code Change over Time”, arXiv:2604.00917, 2026; MSR 2026. Observational study. ↩

39. “To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study”, arXiv:2605.06464, 2026. Preprint. ↩

40. Hao Li et al., “The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering”, 2026. Observational AIDev dataset study. ↩

41. Epoch AI, “What skills does SWE-bench Verified evaluate?”, 2025. ↩

42. “Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub”, arXiv:2601.00477, 2026. Preprint. ↩

43. Carol Liang and Kevin Ho, Stripe, “Can AI agents build real Stripe integrations? We built a benchmark to find out”, 2026-03-02. ↩

44. Addy Osmani, “How to Write a Good Spec for AI Agents”, accessed 2026-08-27. ↩

45. Prithvi Rajasekaran, Anthropic, “Harness design for long-running application development”, 2026-03-24. ↩ ↩²

46. Addy Osmani, “The Orchestration Tax”, accessed 2026-08-27. ↩