Evaluation best practices

Author: OpenAI API Technical Documentation Team
Published: 2026
Source: OpenAI API Docs

Executive Summary & Core Thesis

Generative AI is inherently variable: models produce non-deterministic output from identical inputs, making conventional deterministic software testing insufficient. Evaluations (evals) provide the structured testing discipline required to measure and continuously improve LLM application accuracy, performance, and reliability.

Evals are also the primary prerequisite for systematic performance optimization (prompt engineering, few-shot tuning, and reinforcement fine-tuning).

Core Principles & Workflow

Adopt Eval-Driven Development (EDD): Evaluate early and often; write scoped tests at every phase of development.
Design Task-Specific Evals: Reflect real-world production distribution rather than relying on generic benchmarks.
Log Everything: Instrument comprehensive logging during development to mine production failures for evaluation datasets.
Maintain Human Agreement: Continuously calibrate automated scoring algorithms against expert human annotations.
Continuous Evaluation (CE): Run automated evals on every code/prompt change in CI/CD to detect non-deterministic regressions.

Discriminative Evaluators & LLM-as-a-Judge Guidelines

LLMs excel at discrimination (comparison, scoring, classification) over open-ended generation. Reliable evaluation design aligns with this strength:

Pairwise comparisons & Rubrics: Structure judge tasks as pairwise selections or explicit rubric grading rather than open-ended commentary.
Reasoning-Before-Scoring: Require the judge LLM to generate chain-of-thought reasoning before outputting a numerical score.
Length / Verbosity Bias Correction: Calibrate and control for the LLM judge’s inherent systematic bias toward favoring longer responses.
Separate Judge Model: Utilize distinct, highly capable models for evaluation rather than reusing the generator model.

Edge Case Taxonomy for Production Agents

Input Variability: Multilingual inputs, unstructured formats (XML, JSON, Markdown, CSV), and multi-modal attachments.
Contextual Complexity: Noisy conversation histories, ambiguous tool return keys (e.g. {"on": 123} where on represents order number), multi-intent requests, and circular agent handoffs.
Personalization & Security: System prompt / user prompt conflicts, jailbreak attempts, and strict formatting constraints.

Quantitative Claims & Thresholds

Reference dataset scales: 1,000 reference transcript-summary pairs recommended for held-out summarization evaluation.
Document QA metrics: Target context recall ≥ 0.85, context precision > 0.70, and user satisfaction rate ≥ 70%.
Judge agreement threshold: Scale automated LLM judges only after achieving consistent statistical agreement with human annotations.

Decision Rules

When designing LLM-as-a-judge evaluators, do structure tasks as pairwise comparisons, classification, or strict rubrics rather than open-ended generation, because LLMs are significantly more reliable at discrimination than generation.
When constructing judge prompts, do enforce chain-of-thought reasoning before scoring and control for response length bias, because unconstrained judges systematically favor verbose outputs.
When releasing prompt or pipeline changes, do run continuous evaluation suites against production-mined failure distributions, because localized prompt tweaks frequently cause regressions on unmeasured edge cases.

Anti-Patterns & Failure Modes

Vibe-based evals: Relying on informal subjective impressions (“it looks right”) or postponing eval construction until post-launch.
Overly generic metrics: Relying solely on academic metrics like perplexity, BLEU, or ROUGE in place of task-specific functional tests.
Biased / Synthetic-only datasets: Evaluating exclusively on synthetic happy paths that fail to reproduce noisy production distributions.

What the Source Does NOT Claim

Does NOT claim that LLM judges can operate reliably without calibration against expert human annotations.
Does NOT claim that static benchmark scores guarantee reliability in dynamic multi-turn production environments.