Automated Eval Engineering

Automated eval engineering is the practice of using interactive agent tooling to inspect a software codebase, analyze production execution traces, and generate executable test harnesses and benchmarks for autonomous AI systems.

The Interactive Interview Pattern

As demonstrated in LangChain’s Harbor eval engineering workflows, fully autonomous “one-shot” eval generation frequently produces brittle, low-signal benchmarks or measures trivial properties. The effective pattern centers on an interactive human-in-the-loop interview:

1. Repository Surface Mapping: The eval agent crawls the application repository, cataloging prompts, tools, hooks, skills, schemas, and backing dependencies.
2. Trace Contract Inspection: Analyzing production execution logs (e.g. via langsmith-cli) to observe real tool arguments, outputs, failure states, and external API contracts.
3. Capability Proposals & Human Steering: The agent proposes specific capabilities worth measuring, recommends prioritized directions, and accepts human guidance on which dependencies to run live vs. simulate.
4. Harbor Task Compilation: Synthesizing containerized tasks with four core artifacts:
   • task.toml: metadata and runtime parameters.
   • instruction.md: task prompt delivered to the agent under test.
   • environment/: Dockerfile defining reproducible toolsets and filesystem states.
   • tests/: verifiers scoring agent trajectory, artifacts, and final state.

Evals as Training and Harness Data

Standardized containerized evals decouple the environment from the agent configuration. This allows engineering teams to treat evals as fixed targets while rapidly running parallel sweeps across prompt changes, tool schemas, context engineering strategies, and model fine-tuning.

Related

• error analysis and evals
• closed loop agent improvement
• agentic manual testing
• tdd with agents
• context engineering