AI Product Engineering Notes

Author: Hamel Husain URL: https://hamel.dev/notes/llm/ai-product-engineering/ Date: August 12, 2026

Core Ideas

• AI Product Engineering represents the discipline of transforming raw foundation models into reliable, high-utility products.
• Optimization Hierarchy:
  1. Optimize retrieval, context, and data models first (highest leverage, lowest cost).
  2. Optimize systems, execution sandboxes, and harness engineering next.
  3. Explore model fine-tuning and post-training only after context and harness are exhausted.
• Key Sub-topics & Findings:
  • Error Analysis & Active Learning (Shreya Shankar): Build taxonomies bottom-up from human trace review rather than top-down speculative rubrics.
  • Model Cascades / BARGAIN (Shreya Shankar): Route high-confidence queries to small, cheap models and ambiguous ones to large models, reducing inference cost by up to 86% while maintaining target accuracy.
  • Agent Sandboxes (Adam Azzam / Modal): Coding and research agents need fast, isolated cloud sandbox environments with sub-second spinup times (pre-baked images) and decoupled tool execution architectures to prevent runaway errors from killing trajectories.
  • Harness & Eval Infrastructure Before Post-Training (Prime Intellect): Before training or fine-tuning, verify evaluation harness parameters (e.g. timeout lengths, temperature, turn caps, tool APIs). Benchmarks often swing 15%+ simply from harness fixes.
  • Data Agent Benchmarking (DAB): Multi-hop database agents fail primarily on flawed planning and stubborn adherence to incorrect initial hypotheses rather than data retrieval errors.
  • Search Agents & Synthetic Evaluation (Nandan Thakur / ORBIT & Hawkeye): Synthetic verification loops for multi-hop retrieval; trajectory analysis reveals that correct solutions require significantly fewer search iterations.