Closing the Loop: Evaluating and Improving Replit Agent at Scale

Author: Replit Engineering URL: https://replit.com/blog/evaluating-and-improving-agent-at-scale Date: 2026

Core Ideas

• Agent evaluation must move from a one-way release score gate to a continuous closed loop connecting production telemetry, failure clustering, automated hypothesis generation, PR candidate generation, and benchmark verification.
• ViBench (Vibe Benchmark): Public benchmark measuring whether an agent-built application meets a plain-English Product Requirements Document (PRD). Evaluates end-to-end full-stack app creation from scratch using Playwright-driven natural language test plans and sandboxes.
• Key Empirical Finding: Frontier benchmark scores (e.g. SWE-bench) do not cleanly transfer to full greenfield application generation. Models also degrade when extending their own existing code due to compounding errors.
• Telescope: Production tracing engine that summarizes failure trajectories, embeds them, and clusters failure modes across massive telemetry.
• Self-Improving Candidate Loop: Automatically proposes prompt patches, skill additions, tool fixes, and harness adjustments, opens draft PRs, and validates them against ViBench regression test suites before human signoff.
• Role of Human Taste: Human engineers retain critical control over eval curation (shaping the hill the agent climbs), architecture changes, and final production deployment.