---
title: "AI Product Engineering Notes"
section: "raw"
type: "source"
created: "2026-08-22"
updated: "2026-08-22"
canonical: "https://pyweb.dev/wiki/raw/articles/hamel-ai-product-engineering-notes-2026"
---
# AI Product Engineering Notes

**Author:** Hamel Husain
**URL:** https://hamel.dev/notes/llm/ai-product-engineering/
**Date:** August 12, 2026

## Core Ideas

- AI Product Engineering represents the discipline of transforming raw foundation models into reliable, high-utility products.
- Optimization Hierarchy:
  1. Optimize retrieval, context, and data models first (highest leverage, lowest cost).
  2. Optimize systems, execution sandboxes, and harness engineering next.
  3. Explore model fine-tuning and post-training only after context and harness are exhausted.
- **Key Sub-topics & Findings:**
  - **Error Analysis & Active Learning (Shreya Shankar):** Build taxonomies bottom-up from human trace review rather than top-down speculative rubrics.
  - **Model Cascades / BARGAIN (Shreya Shankar):** Route high-confidence queries to small, cheap models and ambiguous ones to large models, reducing inference cost by up to 86% while maintaining target accuracy.
  - **Agent Sandboxes (Adam Azzam / Modal):** Coding and research agents need fast, isolated cloud sandbox environments with sub-second spinup times (pre-baked images) and decoupled tool execution architectures to prevent runaway errors from killing trajectories.
  - **Harness & Eval Infrastructure Before Post-Training (Prime Intellect):** Before training or fine-tuning, verify evaluation harness parameters (e.g. timeout lengths, temperature, turn caps, tool APIs). Benchmarks often swing 15%+ simply from harness fixes.
  - **Data Agent Benchmarking (DAB):** Multi-hop database agents fail primarily on flawed planning and stubborn adherence to incorrect initial hypotheses rather than data retrieval errors.
  - **Search Agents & Synthetic Evaluation (Nandan Thakur / ORBIT & Hawkeye):** Synthetic verification loops for multi-hop retrieval; trajectory analysis reveals that correct solutions require significantly fewer search iterations.

---

## Agent Navigation

### Machine endpoints
- Knowledge graph: https://pyweb.dev/api/graph.json
- Graph analysis: https://pyweb.dev/api/graph-analysis.json
- Context index: https://pyweb.dev/llms.txt