---
title: "Closing the Loop: Evaluating and Improving Replit Agent at Scale"
section: "raw"
type: "source"
created: "2026-08-22"
updated: "2026-08-22"
canonical: "https://pyweb.dev/wiki/raw/articles/replit-evaluating-and-improving-agent-at-scale-2026"
---
# Closing the Loop: Evaluating and Improving Replit Agent at Scale

**Author:** Replit Engineering
**URL:** https://replit.com/blog/evaluating-and-improving-agent-at-scale
**Date:** 2026

## Core Ideas

- Agent evaluation must move from a one-way release score gate to a continuous closed loop connecting production telemetry, failure clustering, automated hypothesis generation, PR candidate generation, and benchmark verification.
- **ViBench (Vibe Benchmark):** Public benchmark measuring whether an agent-built application meets a plain-English Product Requirements Document (PRD). Evaluates end-to-end full-stack app creation from scratch using Playwright-driven natural language test plans and sandboxes.
- **Key Empirical Finding:** Frontier benchmark scores (e.g. SWE-bench) do not cleanly transfer to full greenfield application generation. Models also degrade when extending their own existing code due to compounding errors.
- **Telescope:** Production tracing engine that summarizes failure trajectories, embeds them, and clusters failure modes across massive telemetry.
- **Self-Improving Candidate Loop:** Automatically proposes prompt patches, skill additions, tool fixes, and harness adjustments, opens draft PRs, and validates them against ViBench regression test suites before human signoff.
- **Role of Human Taste:** Human engineers retain critical control over eval curation (shaping the hill the agent climbs), architecture changes, and final production deployment.

---

## Agent Navigation

### Machine endpoints
- Knowledge graph: https://pyweb.dev/api/graph.json
- Graph analysis: https://pyweb.dev/api/graph-analysis.json
- Context index: https://pyweb.dev/llms.txt