Towards Automating Eval Engineering

Author: LangChain Engineering URL: https://www.langchain.com/blog/towards-automating-eval-engineering Date: July 2026

Core Ideas

• Eval engineering skill is designed to interview the user who can give feedback on proposals and iteratively approve each eval, rather than relying on one-shot automated eval generation.
• The skill maps the agent surface from the repository: prompts, models, tools, skills, hooks, and external backing data/services (e.g. API calls).
• It consumes production traces via tools like langsmith-cli to inspect tool contracts, arguments, results, and runtime errors.
• The end product is containerized Harbor tasks:
  • task.toml configuration
  • instruction.md prompt / start message
  • environment/ Dockerfile setting up tools and filesystem data
  • tests/ verifier checking agent trajectory and end state
• Evals serve as training data and fixed targets for harness engineering (prompt/tool edits) or model fine-tuning.