/agent

wiki / raw / tessl-coding-benchmarks-misaligned-system-harness-2026

Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering

updated 2026-08-27

Original source: https://arxiv.org/abs/2606.17799 SHA256: 4b68e983226a2731dc5e89da3b27b4097495393d2208e4ad9d63c5aa68be3812

Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering

By Maria I. Gorinova, Macey Baker, Amy Heineike, Maksim Shaposhnikov, Rob Willoughby, Dru Knox (Tessl, London, UK). Agentic Software Engineering (SE 3.0) Workshop, ACM SIGKDD 2026 (arXiv:2606.17799v2).

Abstract & Thesis

Coding agents have become a major mode of software engineering, but the benchmarks used to compare them collapse model, harness, and environment into a single end-to-end score against a single reference solution. Current benchmarks are misaligned with agentic software engineering:

  1. Benchmark scores conflate the model with the rest of the harness.
  2. Grading against a single reference solution penalises equally valid alternatives.
  3. The absence of signal at the level of individual harness components makes the end-to-end system score difficult to iterate on.

A coding agent in practice is not a model: it is a system harness — a composite of models, harnesses, contexts, environments, and feedback signals.

The System Harness Architecture

The system harness has five recurring components:

  1. Tasks: units of work derived from higher-level goals.
  2. Agent harnesses: configurable executors composed of model, prompt, tools, and loop.
  3. Environment: repository, runtime under change, issue tracker, CI, deployment surfaces.
  4. Context: curated projection of environment, skills, plugins, hooks, specs.
  5. Feedback signals & Verifiers: signals the harness reads to refine solutions or itself.

Three Tiers of Feedback Signals

Feedback splits into three tiers by scope, latency, and trust:

  • Inner-loop signals (seconds to minutes): tests, types, lint, compilation. Immediate steer on whether the edit is executable, well-typed, and policy-compliant.
  • Middle-loop signals (minutes to hours): reviewer subagent requests, simulation, maintenance agents, score rubrics, mutation checks, and architectural boundary drift.
  • Outer-loop signals (days to weeks): PR acceptance, revert rate, incident reports, production telemetry, customer feedback. Delayed ground truth used to calibrate inner and middle proxies.

NS2 (open-source harness by Tessl) implements this pattern: GitHub issues serve as the coordination state machine, decomposing issues into vertical-slice tasks with multi-tier verification.