--- title: "Evaluation best practices" section: "raw" type: "source" created: "2026-08-27" updated: "2026-08-27" canonical: "https://pyweb.dev/wiki/raw/articles/openai-evaluation-best-practices-2026" --- # Evaluation best practices **Author:** OpenAI API Technical Documentation Team **Published:** 2026 **Source:** [OpenAI API Docs](https://developers.openai.com/api/docs/guides/evaluation-best-practices) --- ## Executive Summary & Core Thesis Generative AI is inherently variable: models produce non-deterministic output from identical inputs, making conventional deterministic software testing insufficient. Evaluations (**evals**) provide the structured testing discipline required to measure and continuously improve LLM application accuracy, performance, and reliability. Evals are also the primary prerequisite for systematic performance optimization (prompt engineering, few-shot tuning, and reinforcement fine-tuning). --- ## Core Principles & Workflow 1. **Adopt Eval-Driven Development (EDD):** Evaluate early and often; write scoped tests at every phase of development. 2. **Design Task-Specific Evals:** Reflect real-world production distribution rather than relying on generic benchmarks. 3. **Log Everything:** Instrument comprehensive logging during development to mine production failures for evaluation datasets. 4. **Maintain Human Agreement:** Continuously calibrate automated scoring algorithms against expert human annotations. 5. **Continuous Evaluation (CE):** Run automated evals on every code/prompt change in CI/CD to detect non-deterministic regressions. --- ## Discriminative Evaluators & LLM-as-a-Judge Guidelines LLMs excel at **discrimination** (comparison, scoring, classification) over open-ended generation. Reliable evaluation design aligns with this strength: - **Pairwise comparisons & Rubrics:** Structure judge tasks as pairwise selections or explicit rubric grading rather than open-ended commentary. - **Reasoning-Before-Scoring:** Require the judge LLM to generate chain-of-thought reasoning before outputting a numerical score. - **Length / Verbosity Bias Correction:** Calibrate and control for the LLM judge's inherent systematic bias toward favoring longer responses. - **Separate Judge Model:** Utilize distinct, highly capable models for evaluation rather than reusing the generator model. --- ## Edge Case Taxonomy for Production Agents 1. **Input Variability:** Multilingual inputs, unstructured formats (XML, JSON, Markdown, CSV), and multi-modal attachments. 2. **Contextual Complexity:** Noisy conversation histories, ambiguous tool return keys (e.g. `{"on": 123}` where `on` represents order number), multi-intent requests, and circular agent handoffs. 3. **Personalization & Security:** System prompt / user prompt conflicts, jailbreak attempts, and strict formatting constraints. --- ## Quantitative Claims & Thresholds - **Reference dataset scales:** 1,000 reference transcript-summary pairs recommended for held-out summarization evaluation. - **Document QA metrics:** Target context recall ≥ **0.85**, context precision > **0.70**, and user satisfaction rate ≥ **70%**. - **Judge agreement threshold:** Scale automated LLM judges only after achieving consistent statistical agreement with human annotations. --- ## Decision Rules - **When designing LLM-as-a-judge evaluators, do structure tasks as pairwise comparisons, classification, or strict rubrics rather than open-ended generation**, because LLMs are significantly more reliable at discrimination than generation. - **When constructing judge prompts, do enforce chain-of-thought reasoning before scoring and control for response length bias**, because unconstrained judges systematically favor verbose outputs. - **When releasing prompt or pipeline changes, do run continuous evaluation suites against production-mined failure distributions**, because localized prompt tweaks frequently cause regressions on unmeasured edge cases. --- ## Anti-Patterns & Failure Modes - **Vibe-based evals:** Relying on informal subjective impressions ("it looks right") or postponing eval construction until post-launch. - **Overly generic metrics:** Relying solely on academic metrics like perplexity, BLEU, or ROUGE in place of task-specific functional tests. - **Biased / Synthetic-only datasets:** Evaluating exclusively on synthetic happy paths that fail to reproduce noisy production distributions. --- ## What the Source Does NOT Claim - Does **NOT** claim that LLM judges can operate reliably without calibration against expert human annotations. - Does **NOT** claim that static benchmark scores guarantee reliability in dynamic multi-turn production environments. --- ## Agent Navigation ### Machine endpoints - Knowledge graph: https://pyweb.dev/api/graph.json - Graph analysis: https://pyweb.dev/api/graph-analysis.json - Context index: https://pyweb.dev/llms.txt