Closed-Loop Agent Improvement

Closed-loop agent improvement is an architectural paradigm for autonomous coding systems where production telemetry, failure clustering, candidate patch generation, and regression benchmarks operate in a continuous cycle under human governance.

Key Subsystems (Replit Architecture)

1. Telemetry Clustering (Telescope): Aggregates millions of user execution trajectories, embeds interaction patterns, and clusters recurring failure modes rather than treating traces as isolated incidents.
2. End-to-End Specification Benchmarking (ViBench): Evaluates whether an agent can build a complete, running web application from a plain-English Product Requirements Document (PRD) from scratch. Utilizes browser automation (Playwright) to exercise multi-step user interactions and state verification in isolated sandboxes.
3. Automated Hypothesis & PR Generation: An engineering agent analyzes failure clusters, drafts targeted patches (prompt adjustments, tool schema repairs, harness enhancements, or new skills), and opens draft pull requests with attached trajectory evidence.
4. Automated Regression Verification: The candidate patch runs against ViBench suites and baseline production trajectories before alerting engineers for human review.

Empirical Findings

• Benchmark Disconnect: High scores on narrow synthetic coding benchmarks (such as SWE-bench) do not reliably predict performance on full greenfield application generation.
• Compounding Self-Extension Errors: Frontier models frequently struggle and compound mistakes when asked to modify or extend their own previously generated codebases.
• The Primacy of Human Taste: Autonomous loops efficiently handle hypothesis testing and mechanical validation, but human engineers remain essential for eval curation (defining what success looks like), architecture shifts, and deployment authorization.

Related

• error analysis and evals
• automated eval engineering
• agentic engineering patterns
• agentic manual testing
• context engineering