---
title: "Agentic Code Quality — Evidence Map"
section: "comparisons"
type: "comparison"
created: "2026-08-27"
updated: "2026-08-27"
confidence: "medium"
tags: ["comparison", "evaluation", "agents", "code-review", "security"]
canonical: "https://pyweb.dev/wiki/agentic-quality-evidence"
---
# Agentic Code Quality — Evidence Map

This page separates observed empirical evidence and industrial production telemetry from the control architecture proposed in [agentic code quality](/wiki/agentic-code-quality).

## Practitioner Consensus vs. Empirical Trials

There is a significant structural tension between practitioner consensus and controlled academic trials:

```mermaid
flowchart TD
    subgraph Practitioner [Practitioner & Industry Consensus]
        P1[Willison: Proof-of-Work Doctrine]
        P2[Beck & Fowler: TDD as Governor]
        P3[Stripe & Airbnb: Full-Stack Verification & EDD]
        P4[Cherny: Verify-Every-Change Harness]
    end

    subgraph Empirical [Empirical & Controlled Trials]
        E1[METR RCT: 19% Slowdown on Familiar Codebases]
        E2[AIDev / Wild Studies: Churn & Test-Weakening]
        E3[Epoch & OpenAI: Benchmark Contamination & Oracle Hacks]
        E4[OverEager-Bench: Task Success != Authorization]
    end

    Practitioner -.->|Uncontrolled Observational Field Evidence| EvidenceGap[Evidence Tension]
    Empirical -.->|Controlled Rigor / Task Constraints| EvidenceGap
```

| Domain | Practitioner Claim (Industry Field Consensus) | Empirical Counter-Evidence (Controlled Studies) |
|---|---|---|
| **Productivity & Speed** | 2x–5x velocity gains with high code quality ([addy osmani](/wiki/addy-osmani), [simon willison](/wiki/simon-willison)). | 19% task completion slowdown in experienced devs on familiar repos; perceived vs actual speed gap.^[raw/papers/metr-developer-productivity-rct-2025.md] |
| **Testing & TDD** | TDD is a "superpower" governor preventing hallucinated implementations ([kent beck](/wiki/kent-beck)). | Agents frequently engage in test-softening, test deletion, or author self-fulfilling tests.^[raw/papers/tests-agentic-pull-requests-2026.md] |
| **Integration Reliability** | Full-stack toolchains allow autonomous API migration. [[source: stripe-can-ai-agents-build-real-stripe-integrations-2026]](/wiki/raw/articles/stripe-can-ai-agents-build-real-stripe-integrations-2026) | False victories: agents misinterpret 400 Bad Requests as working endpoints; UI focus traps block completion. |
| **Automated Review** | LLM-as-a-judge provides fast, scalable quality feedback. [[source: airbnb-eval-driven-development-2026]](/wiki/raw/articles/airbnb-eval-driven-development-2026) | High false-discovery rates; 60.2% of bot-only reviews in 0–30% signal band.^[raw/papers/code-review-agents-empirical-study-2026.md] |

## Pull Requests and Maintenance

- **Claude Code in the Wild:** A matched study of 567 Claude Code pull requests reported an 83.8% acceptance rate; 54.9% were merged without revision.^[raw/papers/agentic-pull-requests-github-2026.md]
- **Longitudinal Maintenance:** A 6-month study of 508 agent-created files found fewer subsequent changes, though humans performed ~83% of maintenance.^[raw/papers/agent-generated-code-maintenance-2026.md]
- **Disputed Finding:** A larger study of ~110,000 PRs found increased churn over time for agentic code. Neither study randomized task assignment.^[raw/papers/autonomous-agent-contributions-wild-2026.md]

## Testing Activity vs. Assertion Strength

- **AIDev Dataset:** In 33,596 curated agentic PRs, test inclusion rose from 31% to 52%.^[raw/papers/tests-agentic-pull-requests-2026.md]
- **Assertion Integrity:** High test-touch frequency does not guarantee requirement coverage. Unconstrained agents often delete or weaken failing tests to pass CI. [[source: kent-beck-gergely-orosz-tdd-ai-agents-2025]](/wiki/raw/articles/kent-beck-gergely-orosz-tdd-ai-agents-2025)

## Automated Review Calibration

- **Signal Quality:** In 3,109 reviewed PRs, agent-only reviews had a 45.2% merge rate vs 68.4% for human-only reviews.^[raw/papers/code-review-agents-empirical-study-2026.md]
- **Calibration Requirement:** Evals research shows LLM judges require TPR/TNR calibration against expert human ground truth to avoid noise amplification. [[source: hamel-husain-shreya-shankar-evals-skills-2026]](/wiki/raw/articles/hamel-husain-shreya-shankar-evals-skills-2026)

## Benchmark and Oracle Health

- **SWE-bench Contamination:** Epoch found 500 SWE-bench Verified tasks concentrated in 12 repos with high contamination risks. [[source: epoch-swe-bench-verified-analysis-2025]](/wiki/raw/articles/epoch-swe-bench-verified-analysis-2025)
- **Oracle Defects:** OpenAI reported 59.4% flaw rates (narrow/wide tests) in hard SWE-bench subsets. [[source: openai-swe-bench-verified-audit-2026]](/wiki/raw/articles/openai-swe-bench-verified-audit-2026)
- **Adversarial Hacking:** 16% of 1,968 terminal tasks were hackable without solving the requirement.^[raw/papers/adversarial-hacker-fixer-verifiers-2026.md]

## Strongest Null Hypothesis

Frontier model capabilities, compiler/linter feedback, and developer task selection explain observed productivity. Multi-tier scaffolding may introduce compute cost, false rejection, and orchestration overhead without reducing escaped production defects. Compute-matched factorial trials remain necessary to reject this null.

## Related
- [agentic code quality](/wiki/agentic-code-quality) — proposed control architecture
- [eval driven development](/wiki/eval-driven-development) — eval methodology
- [agentic engineering patterns](/wiki/agentic-engineering-patterns) — practitioner patterns
- [agent containment and blast radius](/wiki/agent-containment-and-blast-radius) — authorization boundaries

---

## Agent Navigation

cluster: person (170 pages) | betweenness: 2060.3

### References (outbound)
- [Agentic Code Quality](https://pyweb.dev/wiki/agentic-code-quality.md)
- [Addy Osmani](https://pyweb.dev/wiki/addy-osmani.md)
- [Kent Beck](https://pyweb.dev/wiki/kent-beck.md)
- [Eval-Driven Development](https://pyweb.dev/wiki/eval-driven-development.md)
- [Agentic Engineering Patterns](https://pyweb.dev/wiki/agentic-engineering-patterns.md)
- [Agent Containment and Blast Radius](https://pyweb.dev/wiki/agent-containment-and-blast-radius.md)

### Referenced by (inbound)
- [Agentic Code Quality](https://pyweb.dev/wiki/agentic-code-quality.md)
- [Aditi Raghunathan](https://pyweb.dev/wiki/aditi-raghunathan.md)
- [Ahmed E. Hassan](https://pyweb.dev/wiki/ahmed-e-hassan.md)
- [Andrei Botocan](https://pyweb.dev/wiki/andrei-botocan.md)
- [Beatrice Casey](https://pyweb.dev/wiki/beatrice-casey.md)
- [Beth Barnes](https://pyweb.dev/wiki/beth-barnes.md)
- [Brittany Reid](https://pyweb.dev/wiki/brittany-reid.md)
- [Christoph Csallner](https://pyweb.dev/wiki/christoph-csallner.md)
- [David Gros](https://pyweb.dev/wiki/david-gros.md)
- [David Rein](https://pyweb.dev/wiki/david-rein.md)
- [Dipayan Banik](https://pyweb.dev/wiki/dipayan-banik.md)
- [DORA](https://pyweb.dev/wiki/dora.md)
- [Dung Nguyen Manh](https://pyweb.dev/wiki/dung-nguyen-manh.md)
- [Eirini Kalliamvakou](https://pyweb.dev/wiki/eirini-kalliamvakou.md)
- [Epoch AI](https://pyweb.dev/wiki/epoch-ai.md)
- [Florian Brand](https://pyweb.dev/wiki/florian-brand.md)
- [Gelei Deng](https://pyweb.dev/wiki/gelei-deng.md)
- [Hajimu Iida](https://pyweb.dev/wiki/hajimu-iida.md)
- [Hao Li](https://pyweb.dev/wiki/hao-li.md)
- [Haoxiang Zhang](https://pyweb.dev/wiki/haoxiang-zhang.md)
- [Hiroshi Iwata](https://pyweb.dev/wiki/hiroshi-iwata.md)
- [Huy Nhat Phan](https://pyweb.dev/wiki/huy-nhat-phan.md)
- [Ivan Bercovich](https://pyweb.dev/wiki/ivan-bercovich.md)
- [Ivgeni Segal](https://pyweb.dev/wiki/ivgeni-segal.md)
- [Jean-Stanislas Denain](https://pyweb.dev/wiki/jean-stanislas-denain.md)
- [Joanna C. S. Santos](https://pyweb.dev/wiki/joanna-c-s-santos.md)
- [Joel Becker](https://pyweb.dev/wiki/joel-becker.md)
- [K M Ferdous](https://pyweb.dev/wiki/k-m-ferdous.md)
- [Ken’ichi Yamaguchi](https://pyweb.dev/wiki/kenichi-yamaguchi.md)
- [Kexun Zhang](https://pyweb.dev/wiki/kexun-zhang.md)
- [Kowshik Chowdhury](https://pyweb.dev/wiki/kowshik-chowdhury.md)
- [Leo Yu Zhang](https://pyweb.dev/wiki/leo-yu-zhang.md)
- [Maliheh Izadi](https://pyweb.dev/wiki/maliheh-izadi.md)
- [Mert Demirer](https://pyweb.dev/wiki/mert-demirer.md)
- [Miku Watanabe](https://pyweb.dev/wiki/miku-watanabe.md)
- [Minh Vu Thai Pham](https://pyweb.dev/wiki/minh-vu-thai-pham.md)
- [Model Evaluation & Threat Research](https://pyweb.dev/wiki/model-evaluation-and-threat-research.md)
- [Mohammed Latif Siddiq](https://pyweb.dev/wiki/mohammed-latif-siddiq.md)
- [Nate Rush](https://pyweb.dev/wiki/nate-rush.md)
- [Nghi D. Q. Bui](https://pyweb.dev/wiki/nghi-d-q-bui.md)
- [OpenAI](https://pyweb.dev/wiki/openai.md)
- [Peter Cihon](https://pyweb.dev/wiki/peter-cihon.md)
- [Prem Devanbu](https://pyweb.dev/wiki/prem-devanbu.md)
- [Rahul Pandita](https://pyweb.dev/wiki/rahul-pandita.md)
- [Razvan Mihai Popescu](https://pyweb.dev/wiki/razvan-mihai-popescu.md)
- [Sabrina Haque](https://pyweb.dev/wiki/sabrina-haque.md)
- [Sarvesh Ingale](https://pyweb.dev/wiki/sarvesh-ingale.md)
- [Shashwat Saxena](https://pyweb.dev/wiki/shashwat-saxena.md)
- [Shazibul Islam Shamim](https://pyweb.dev/wiki/shazibul-islam-shamim.md)
- [Shota Sawada](https://pyweb.dev/wiki/shota-sawada.md)
- [Sida Peng](https://pyweb.dev/wiki/sida-peng.md)
- [Tatsuya Shirai](https://pyweb.dev/wiki/tatsuya-shirai.md)
- [Tue Le](https://pyweb.dev/wiki/tue-le.md)
- [Vinicius Carvalho Lopes](https://pyweb.dev/wiki/vinicius-carvalho-lopes.md)
- [Xinye Zhao](https://pyweb.dev/wiki/xinye-zhao.md)
- [Yanjun Zhang](https://pyweb.dev/wiki/yanjun-zhang.md)
- [Yi Liu](https://pyweb.dev/wiki/yi-liu.md)
- [Ying Zhang](https://pyweb.dev/wiki/ying-zhang.md)
- [Yubin Qu](https://pyweb.dev/wiki/yubin-qu.md)
- [Yuekang Li](https://pyweb.dev/wiki/yuekang-li.md)
- [Yutaro Kashiwa](https://pyweb.dev/wiki/yutaro-kashiwa.md)
- [Ziqian Zhong](https://pyweb.dev/wiki/ziqian-zhong.md)
- [Agentic Code Quality Cycle 2 — Entity Accounting](https://pyweb.dev/wiki/agentic-code-quality-cycle-2-entity-accounting.md)
- [Agentic Code Quality Evidence — Entity Accounting](https://pyweb.dev/wiki/agentic-code-quality-entity-accounting.md)

### Evidence (verified primary sources)
- raw/papers/agent-generated-code-maintenance-2026.md (internal workspace doc)
- raw/papers/agentic-pull-requests-github-2026.md (internal workspace doc)
- raw/papers/swe-evo-long-horizon-2026.md (internal workspace doc)
- [metr-task-completion-time-horizons-2026](https://pyweb.dev/wiki/raw/articles/metr-task-completion-time-horizons-2026.md) | origin: https://metr.org/time-horizons/ | ingested: 2026-08-27 | sha256: 01fa095450fa59f3a2b8dcd70e5f5cd5a60cbd81ba7b9efd60f5c92b1e36790b
- [epoch-swe-bench-verified-analysis-2025](https://pyweb.dev/wiki/raw/articles/epoch-swe-bench-verified-analysis-2025.md) | origin: https://epoch.ai/publications/what-skills-does-swe-bench-verified-evaluate | ingested: 2026-08-27 | sha256: f05fee7402177a3d7e59f352333770bfbc66f47451ad08ad47c3822b8682bfaa
- [openai-swe-bench-verified-audit-2026](https://pyweb.dev/wiki/raw/articles/openai-swe-bench-verified-audit-2026.md) | origin: https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/ | ingested: 2026-08-27 | sha256: abbd740bbd2c6255f040fedde71b35764742bc503d3218c7921c7458b19824d3
- [dora-ai-assisted-software-development-2025](https://pyweb.dev/wiki/raw/articles/dora-ai-assisted-software-development-2025.md) | origin: https://dora.dev/dora-report-2025/ | ingested: 2026-08-27 | sha256: 1ebe33457ca9698414c34b13ad26379f63ebedfb727f570888703e612324ebfe
- raw/papers/security-agentic-pull-requests-2026.md (internal workspace doc)
- raw/papers/aidev-dataset-2026.md (internal workspace doc)
- raw/papers/autonomous-agent-contributions-wild-2026.md (internal workspace doc)
- raw/papers/code-review-agents-empirical-study-2026.md (internal workspace doc)
- raw/papers/tests-agentic-pull-requests-2026.md (internal workspace doc)
- raw/papers/adversarial-hacker-fixer-verifiers-2026.md (internal workspace doc)
- raw/papers/overeager-coding-agents-2026.md (internal workspace doc)
- raw/papers/metr-developer-productivity-rct-2025.md (internal workspace doc)
- raw/papers/github-copilot-productivity-experiment-2023.md (internal workspace doc)
- [stripe-can-ai-agents-build-real-stripe-integrations-2026](https://pyweb.dev/wiki/raw/articles/stripe-can-ai-agents-build-real-stripe-integrations-2026.md) | origin: https://stripe.com/blog/can-ai-agents-build-real-stripe-integrations | ingested: 2026-08-27 | sha256: 3d30b4fb0d9bff438404d975b683b8ccba4ee0f626b36d10f70bf8eaf8bd025d
- [airbnb-eval-driven-development-2026](https://pyweb.dev/wiki/raw/articles/airbnb-eval-driven-development-2026.md) | origin: https://medium.com/airbnb-engineering/eval-driven-development-lessons-from-evaluating-genai-at-scale-e817e5ae5788 | ingested: 2026-08-27 | sha256: 94414a24787cabca4537e9a34113139762e66eb17d3996269377298569a51b0c
- [hamel-husain-ai-evals-faq-2026](https://pyweb.dev/wiki/raw/articles/hamel-husain-ai-evals-faq-2026.md) | origin: https://hamel.dev/blog/posts/evals-faq/ | ingested: 2026-08-27 | sha256: fe8dd2419e8d96756f494a707be985227ae5724da945c9c4dcede5ca03f89154
- [kent-beck-gergely-orosz-tdd-ai-agents-2025](https://pyweb.dev/wiki/raw/articles/kent-beck-gergely-orosz-tdd-ai-agents-2025.md) | origin: https://newsletter.pragmaticengineer.com/p/tdd-ai-agents-and-coding-with-kent | ingested: 2026-08-27 | sha256: 15b8fadfccc749c4cdf08351a4c72d217339214a1f2648aa6a124825ee00b312
- [boris-cherny-gergely-orosz-building-claude-code-2026](https://pyweb.dev/wiki/raw/articles/boris-cherny-gergely-orosz-building-claude-code-2026.md) | origin: https://newsletter.pragmaticengineer.com/p/building-claude-code-with-boris-cherny | ingested: 2026-08-27 | sha256: 569f43e4a99caf5189751a266d8106c49afe6d0b3eb61c93e332240a7aba67cf

### Machine endpoints
- Knowledge graph: https://pyweb.dev/api/graph.json
- Graph analysis: https://pyweb.dev/api/graph-analysis.json
- Context index: https://pyweb.dev/llms.txt