--- title: "You can't whisper at an AI agent: What a dozen steering experiments taught us about how LLM-based agents actually behave in the wild" section: "raw" type: "source" created: "2026-08-27" updated: "2026-08-27" canonical: "https://pyweb.dev/wiki/raw/articles/stripe-you-cant-whisper-at-an-ai-agent-2026" --- # You can't whisper at an AI agent: What a dozen steering experiments taught us about how LLM-based agents actually behave in the wild **Authors:** James Beswick, Peter Epsteen (Stripe Developer Relations & Growth Engineering) **Published:** 2026-05-14 **Source:** [Stripe.dev Blog](https://stripe.dev/blog/ai-steering-experiments) --- ## Executive Summary & Core Thesis Developers often hold a flawed mental model that AI agents behave like diligent junior engineers who read documentation, browse project dependencies, notice deprecation warnings, and ask clarifying questions when stuck. Stripe ran roughly a dozen steering experiments to understand how to make AI agents adopt current API versions, follow integration best practices, and leverage official agent skills. The results demonstrated that **passive hints consistently fail**, while **active prompts and distribution plays succeed**. Agents do not wander or explore; they operate like "contractors on a deadline" optimizing strictly for immediate task completion. --- ## Taxonomy of Steering Experiments 1. **Passive hints (Failed):** - Warning hashes (`warn` objects) in API JSON responses. - Steering cues, READMEs, inline comments, and `AGENTS.md` files placed in dependency package roots. 2. **Active prompts (Succeeded):** - Progressive disclosure in skill files (modular on-demand sub-skills vs monolithic files). - CLI help output optimizations and explicit blocking errors in API compatibility mode. 3. **Distribution plays (Succeeded):** - Skill promotion on `stripe login` confirmation screens. - Skill install buttons and direct paths on `docs.stripe.com`. --- ## Named Frameworks & Patterns (Verbatim) - **"Hard steering vs soft steering"**: The foundational design axis for agent infrastructure. Hard steers (blocking errors, required context in prompt) enforce compliance; soft steers (warnings, hints, adjacent files) are ignored as noise. - **"Progressive disclosure in skill files"**: Structuring agent skill definitions as modular, specialized sub-skills loaded on-demand rather than monolithic files. - **"Contractor on a deadline"**: The operational model of agent behavior—agents do not browse, explore, or seek comprehension; they execute the immediate path to task closure. - **"Distribution problem, not a content problem"**: Agent-facing developer experience succeeds or fails on context delivery (awareness → install → load → follow), not documentation prose quality. - **"Lost in the middle"**: Attention decay in LLMs over long monolithic context files where middle tokens suffer degraded retrieval. --- ## Quantitative Claims & Qualifiers - **Progressive disclosure advantage:** Modular skill files outperformed monolithic blobs by **roughly 10%** across Stripe's eval suite while meaningfully reducing token consumption. - **CLI login conversion rate:** Embedding a one-click `npx` skill install prompt on the CLI login confirmation page yielded a **30–35% conversion rate**. - **Passive hint efficacy:** 0% effective response—agents completely ignored `warn` hashes in API responses and `AGENTS.md` in dependency folders. --- ## Decision Rules - **When designing developer interfaces and APIs for AI agents, do use hard steers (blocking errors, explicit loaded context) rather than soft steers (warnings, hints, documentation files)**, because agents treat non-blocking warnings as noise and optimize strictly for task completion. - **When authoring agent skills and tool instructions, do use progressive disclosure (modular sub-skills)**, because it improves benchmark performance by ~10%, reduces token consumption, and prevents "lost in the middle" attention decay across long contexts. - **When steering agents toward current API versions, do fail hard with explicit blocking compatibility errors**, because agents reliably detect and correct blocking errors but ignore soft deprecation warnings in response headers. - **When distributing agent skills to developers, do place actionable install prompts at high-intent CLI checkpoints (e.g., login confirmation screens)**, because conversion rates can reach 30–35% when presented at moments of high intent. --- ## Anti-Patterns & Failure Modes (with Mechanisms) ### 1. SDK / Dependency Guidance (`AGENTS.md` in Packages) - **Mechanism:** Embedding `AGENTS.md` or comments inside SDK dependency roots under the assumption that agents inspect project dependencies. - **Observable Symptom:** Agents never navigate into `node_modules` or library source trees during task execution; the guidance remains completely unread and ignored. ### 2. API Response Soft Warnings (`warn` Hashes) - **Mechanism:** Returning structured warning objects or deprecation notices alongside successful HTTP 200 responses. - **Observable Symptom:** The agent extracts the specific data fields required for its immediate goal and discards the warning payload entirely, perpetuating deprecated API usage. ### 3. Monolithic Skill Files - **Mechanism:** Placing exhaustive platform documentation and all API domain guides into a single large skill file. - **Observable Symptom:** The LLM suffers attention decay ("lost in the middle"), wastes context window budget, and exhibits a 10% lower success rate compared to modular architectures. --- ## What the Source Does NOT Claim - Does **NOT** claim that high-quality human documentation is unnecessary; rather, human-facing DX is forgiving while agent-facing DX requires strict active context injection. - Does **NOT** claim that agents possess intrinsic motivation to refactor or modernize codebases unless explicitly blocked by runtime errors or contractual prompts. --- ## Agent Navigation ### Machine endpoints - Knowledge graph: https://pyweb.dev/api/graph.json - Graph analysis: https://pyweb.dev/api/graph-analysis.json - Context index: https://pyweb.dev/llms.txt