/agent

wiki / concepts / designing-for-verifiability

Designing for Verifiability

high confidence updated 2026-08-30 evaluation · workflow · feedback-loops · principle

Designing for Verifiability

hamel husain argues that the most common objection to evals — “our product is hard to eval” — is a product smell. Artifacts that are hard for the builder to verify are often hard for users too; in the worst case users must redo the work from scratch to check the output. Designing the product for ease of verification should come before building evals. [source: hamel-husain-it-s-hard-to-eval-is-a-product-smell-2026]

output = answer
       + metric definition      (what exactly was measured)
       + trusted comparison     (vetted baseline)
       + sanity checks          (distribution, related numbers)
       + what could NOT be verified

The anti-pattern: answer-only output

An AI data agent that returns only “Net revenue for Product A last quarter was $4.21M” gives the user nothing to check. Since the only output is the answer, verification means redoing the analysis.

The fix: checkable artifacts

Design outputs around how a domain expert would validate them. For a metrics agent, that means surfacing:

  • Comparison against a trusted source (a vetted dashboard or a colleague’s prior analysis)
  • The precise metric definition used (does “net revenue” net out returns and discounts?)
  • Sanity checks on related quantities and the distribution beneath the aggregate
  • The query itself, runnable and editable
  • An explicit “what I could not verify” section — flag unverified inputs instead of presenting them as settled [source: hamel-husain-it-s-hard-to-eval-is-a-product-smell-2026]

A two-level interface serves both audiences: a chat reply surfacing the details worth seeing up front, backed by a full notebook holding the complete analysis.

Why this matters for agents

Verifiability-first design is the product-side twin of the harness-side principle that agents need deterministic quality gates (agent harness engineering). Both reduce to the same move: make correctness checkable by construction, rather than asserting it. It also makes evals tractable — checkable artifacts are exactly the surfaces an eval can grade.

Rule of Thumb

If verifying the output takes longer than redoing the work, the output is not done - design artifacts users can check, not answers they must trust.

Failure Modes

SymptomRoot causeFix
User redoes the analysis to trust the answerAnswer-only outputShip checkable artifacts: query, definition, comparison, unknowns
Trust decays over sessionsNo consistency baselineCompare against a vetted source on every run
Evidence — verified primary sources
hamel-husain-it-s-hard-to-eval-is-a-product-smell-2026 https://hamel.dev/blog/posts/eval-smell/
ingested 2026-08-24
sha256:bf1a77e2fc29…
Graph context cluster: person (170 pages) betweenness: 218.9 (rank #23)
References (6)
Hamel Husainargues that the most common objection to evals — "our product is hard to eval" — is a product smell. Artifacts that are hard for the builderAgent Harness Engineering). Both reduce to the same move: make correctness checkable by construction, rather than asserting it. It also makes evals tractable — checkError Analysis and EvalsAutomated Eval EngineeringAgentic Manual TestingTDD With Agents
Referenced by (6)
Agentic Code Qualityinspectable product contractsError Analysis and EvalsEvals SkillsSite-as-Eval-Subject Gradingapplies to theHamel Husain).^[raw/articles/hamel-husain-it-s-hard-to-eval-is-a-product-smell-2026.md]Tesslobjective verification criteria over reference solutions