loading…

Agentic Code Quality

Agentic code quality is usefully modeled as a property of the model–scaffold–toolchain–policy system, not the generator alone. While practitioners emphasize patterns like agentic engineering patterns and eval driven development, empirical research reveals distinct failure modes: test-oracle defects, reward hacking, scope expansion, noisy automated review, and long-horizon degradation. agentic quality evidence separates observed evidence from the architecture proposed here.

Quality Vector

A software factory should report independent acceptance dimensions plus their joint intersection:

Dimension	Direct evidence	Proxy to distrust
Requirement correctness	Hidden behavioral tests plus expert adjudication	Visible tests alone
Regression safety	Full-suite, differential, property, or metamorphic checks	Changed tests only
Security	Executable abuse cases and confirmed findings	Scanner count
Architecture	Dependency contracts around stable public boundaries	Folder naming
Test integrity	Protected baselines and mutation survival	Raw coverage
Scope discipline	Audited actions against an authorization allowlist	Task completion
Reviewability	True findings per reviewer minute (TPR/TNR calibrated)	Comment volume
Operability	Canary health, rollback, defect survival	Merge success
Economics	Cost and elapsed time per releasable change	LOC or PR count

At the factory exit, report correctness_pass, regression_pass, security_pass, architecture_pass, and scope_pass, plus their joint releasable patch rate. Critical security, authorization, and data-loss events remain separate tail outcomes.

Risk-Conditioned Control Architecture

flowchart TD
    I[Human Intent and Authorization] --> S[Executable Spec and Contracts]
    S --> C[Context Assembly]
    C --> G[Generator in Isolated Worktree]
    G --> F[Fast Deterministic Checks]
    F -->|Fail: bounded repair| G
    F --> B[Behavior and Regression Oracles]
    B -->|Fail: bounded repair| G
    B --> A[Architecture and Security Contracts]
    A -->|Fail: bounded repair| G
    A --> R[Calibrated Independent Review]
    R -->|Actionable finding: bounded repair| G
    R --> H{Risk Gate}
    H -->|Low risk and observable proof| D[Canary Deployment]
    H -->|Ambiguity or high blast radius| U[Accountable Human Decision]
    D --> T[Telemetry and Rollback]
    T --> S

Tier 0 — Authorization and Containment

• Declare allowed repositories, paths, services, secrets, and deployment targets.
• Mount hidden conformance suites and protected regression baselines read-only outside the generator worktree.
• Keep candidate test paths writable so the generator can use red green tdd; evaluate those tests independently.
• Isolate filesystem, processes, credentials, and network egress to bound blast radius.^[raw/papers/overeager-coding-agents-2026.md]
• Record tool actions through audit channels the agent cannot bypass.

Tier 1 — Fast Inner Loop

Run after each coherent edit:

• TypeScript: oxlint for fast AST checks (see deterministic lint gates for the error-only gate pattern); tsc --noEmit --incremental for type compatibility; changed Vitest/Jest unit tests. [source: oxlint-type-aware-linting-2026]
• Python: Ruff lint/format; Pyright or mypy under repo policy; changed-scope pytest. Fast checks create cheap back-pressure by catching local syntax and type impossibilities.

Tier 2 — Structural and Behavioral Contracts

Before review:

• Hidden acceptance tests derived from requirements; full regression tests for affected dependency cones.
• Property, differential, or metamorphic checks where suitable; API and schema contracts.
• Graph conformance: dependency-cruiser rules for JavaScript/TypeScript and import-linter layer contracts for Python. [source: dependency-cruiser-rules-reference-2026] [source: import-linter-layer-contracts-2026]

Tier 3 — Test Strength and Adversarial Verification

Used selectively for critical domain, parser, financial, cryptographic, or security logic:

• Mutation testing via StrykerJS or mutmut to measure assertion strength. [source: strykerjs-configuration-2026] [source: mutmut-documentation-2026]
• Adversarial probes: malformed input fuzzing and hacker–fixer–solver loops against acceptance verifiers.^[raw/papers/adversarial-hacker-fixer-verifiers-2026.md]

Tier 4 — Calibrated Independent Review

• Independent judge agents review diffs against specifications, non-goals, and security policies before seeing implementer rationale.
• Reviewer agents must be calibrated on expert-labeled sets, reporting true positive (TPR) and true negative rates (TNR) to avoid low-signal review noise.^[raw/papers/code-review-agents-empirical-study-2026.md] [source: hamel-husain-shreya-shankar-evals-skills-2026]

Tier 5 — Risk Gate and Production Feedback

Auto-merge only for pre-approved low-risk changes meeting observable criteria:

• Conformance suite passes; touched resources match allowlist; regression checks pass without exceptions.
• Canary monitoring and rollback triggers active. Ambiguous intent or high blast radius requires accountable human sign-off.

Controls by Task Risk

Task	Minimum control	Human role
Documentation or isolated formatting	T0–T1 plus link/build checks	Sample audit
Local bug fix with reproducer	T0–T2 plus protected regression	Review exceptions
Feature in one bounded module	T0–T2 plus API/property checks	Approve spec or boundary changes
Core domain, parser, financial, crypto, auth	T0–T3 with mutation/adversarial checks	Mandatory accountable review
Cross-module refactor	T0–T4 with dependency graph & characterization	Review architecture & migration
Platform or data migration	T0–T5 with compatibility matrix & canary	Approve rollout
Ambiguous product behavior	Prototype and clarify first	Decide intent; no auto-merge

Oracle Health

A green verifier can still be flawed. Epoch and OpenAI audits demonstrated SWE-bench task contamination, test over-specificity, and exploitable test harness bugs. [source: epoch-swe-bench-verified-analysis-2025] [source: openai-swe-bench-verified-audit-2026] Production harnesses must treat evaluation oracles as production software subject to regression suites, mutation checks, and eval driven development.

What Remains Unverified

• The causal effect of the complete multi-tier stack on escaped production defects.
• Long-term maintenance cost under task-matched random assignment.
• Mutation testing’s cost-effectiveness as an autonomous merge gate.
• The strongest null: frontier model capability, basic compiler/linter feedback, and human task selection may drive nearly all observed success; multi-tier scaffolding may add compute and false rejection without reducing escapes.

Related

• five debts of agentic engineering — generative debt modes and control mappings
• releasable patch rate — north-star factory metric
• constraint layering — allocating controls across prompt, tools, and sandboxes
• skill treatment effect — empirical evaluation of procedural skills
• agentic quality evidence — empirical evidence and contradictions
• eval driven development — error-discovery eval loop
• agentic engineering patterns — disciplined practitioner patterns
• agentic software factory — operating model
• agent containment and blast radius — authorization and damage bounds
• designing for verifiability — inspectable product contracts
• agentic code quality entity accounting — cycle 1 source-author dispositions
• agentic code quality cycle 2 entity accounting — cycle 2 source-author dispositions