---
title: "Designing for Verifiability"
description: "'It's hard to eval' is a product smell — artifacts hard for you to verify are hard for users too; design checkable artifacts before building evals."
section: "concepts"
type: "concept"
created: "2026-08-24"
updated: "2026-08-30"
confidence: "high"
tags: ["evaluation", "workflow", "feedback-loops", "principle"]
canonical: "https://pyweb.dev/wiki/designing-for-verifiability"
---
# Designing for Verifiability

[hamel husain](/wiki/hamel-husain) argues that the most common objection to evals — "our product is hard to eval" — is a **product smell**. Artifacts that are hard for the builder to verify are often hard for users too; in the worst case users must redo the work from scratch to check the output. Designing the product for ease of verification should come *before* building evals. [[source: hamel-husain-it-s-hard-to-eval-is-a-product-smell-2026]](/wiki/raw/articles/hamel-husain-it-s-hard-to-eval-is-a-product-smell-2026)


```text
output = answer
       + metric definition      (what exactly was measured)
       + trusted comparison     (vetted baseline)
       + sanity checks          (distribution, related numbers)
       + what could NOT be verified
```

## The anti-pattern: answer-only output

An AI data agent that returns only "Net revenue for Product A last quarter was $4.21M" gives the user nothing to check. Since the only output is the answer, verification means redoing the analysis.

## The fix: checkable artifacts

Design outputs around how a domain expert would validate them. For a metrics agent, that means surfacing:

- Comparison against a **trusted source** (a vetted dashboard or a colleague's prior analysis)
- The precise **metric definition** used (does "net revenue" net out returns and discounts?)
- **Sanity checks** on related quantities and the distribution beneath the aggregate
- The **query itself**, runnable and editable
- An explicit **"what I could not verify"** section — flag unverified inputs instead of presenting them as settled [[source: hamel-husain-it-s-hard-to-eval-is-a-product-smell-2026]](/wiki/raw/articles/hamel-husain-it-s-hard-to-eval-is-a-product-smell-2026)

A two-level interface serves both audiences: a chat reply surfacing the details worth seeing up front, backed by a full notebook holding the complete analysis.

## Why this matters for agents

Verifiability-first design is the product-side twin of the harness-side principle that agents need deterministic quality gates ([agent harness engineering](/wiki/agent-harness-engineering)). Both reduce to the same move: make correctness checkable by construction, rather than asserting it. It also makes evals tractable — checkable artifacts are exactly the surfaces an eval can grade.

## Rule of Thumb

If verifying the output takes longer than redoing the work, the output
is not done - design artifacts users can check, not answers they must
trust.

## Failure Modes

| Symptom | Root cause | Fix |
|---|---|---|
| User redoes the analysis to trust the answer | Answer-only output | Ship checkable artifacts: query, definition, comparison, unknowns |
| Trust decays over sessions | No consistency baseline | Compare against a vetted source on every run |

## Related Concepts
- [error analysis and evals](/wiki/error-analysis-and-evals)
- [automated eval engineering](/wiki/automated-eval-engineering)
- [agentic manual testing](/wiki/agentic-manual-testing)
- [tdd with agents](/wiki/tdd-with-agents)
- [hamel husain](/wiki/hamel-husain)

---

## Agent Navigation

cluster: person (170 pages) | betweenness: 218.9

### References (outbound)
- [Hamel Husain](https://pyweb.dev/wiki/hamel-husain.md)
- [Agent Harness Engineering](https://pyweb.dev/wiki/agent-harness-engineering.md)
- [Error Analysis and Evals](https://pyweb.dev/wiki/error-analysis-and-evals.md)
- [Automated Eval Engineering](https://pyweb.dev/wiki/automated-eval-engineering.md)
- [Agentic Manual Testing](https://pyweb.dev/wiki/agentic-manual-testing.md)
- [TDD With Agents](https://pyweb.dev/wiki/tdd-with-agents.md)

### Referenced by (inbound)
- [Agentic Code Quality](https://pyweb.dev/wiki/agentic-code-quality.md)
- [Error Analysis and Evals](https://pyweb.dev/wiki/error-analysis-and-evals.md)
- [Evals Skills](https://pyweb.dev/wiki/evals-skills.md)
- [Site-as-Eval-Subject Grading](https://pyweb.dev/wiki/site-as-eval-subject-grading.md)
- [Hamel Husain](https://pyweb.dev/wiki/hamel-husain.md)
- [Tessl](https://pyweb.dev/wiki/tessl.md)

### Evidence (verified primary sources)
- [hamel-husain-it-s-hard-to-eval-is-a-product-smell-2026](https://pyweb.dev/wiki/raw/articles/hamel-husain-it-s-hard-to-eval-is-a-product-smell-2026.md) | origin: https://hamel.dev/blog/posts/eval-smell/ | ingested: 2026-08-24 | sha256: bf1a77e2fc299c4d3d2489ff9800774406e8d475eb54f9578917f56fe4a7543c

### Machine endpoints
- Knowledge graph: https://pyweb.dev/api/graph.json
- Graph analysis: https://pyweb.dev/api/graph-analysis.json
- Context index: https://pyweb.dev/llms.txt