---
title: "Towards Automating Eval Engineering"
section: "raw"
type: "source"
created: "2026-08-22"
updated: "2026-08-22"
canonical: "https://pyweb.dev/wiki/raw/articles/langchain-automating-eval-engineering-2026"
---
# Towards Automating Eval Engineering

**Author:** LangChain Engineering
**URL:** https://www.langchain.com/blog/towards-automating-eval-engineering
**Date:** July 2026

## Core Ideas

- Eval engineering skill is designed to interview the user who can give feedback on proposals and iteratively approve each eval, rather than relying on one-shot automated eval generation.
- The skill maps the agent surface from the repository: prompts, models, tools, skills, hooks, and external backing data/services (e.g. API calls).
- It consumes production traces via tools like `langsmith-cli` to inspect tool contracts, arguments, results, and runtime errors.
- The end product is containerized Harbor tasks:
  - `task.toml` configuration
  - `instruction.md` prompt / start message
  - `environment/` Dockerfile setting up tools and filesystem data
  - `tests/` verifier checking agent trajectory and end state
- Evals serve as training data and fixed targets for harness engineering (prompt/tool edits) or model fine-tuning.

---

## Agent Navigation

### Machine endpoints
- Knowledge graph: https://pyweb.dev/api/graph.json
- Graph analysis: https://pyweb.dev/api/graph-analysis.json
- Context index: https://pyweb.dev/llms.txt