loading…

You can’t whisper at an AI agent: What a dozen steering experiments taught us about how LLM-based agents actually behave in the wild

Authors: James Beswick, Peter Epsteen (Stripe Developer Relations & Growth Engineering)
Published: 2026-05-14
Source: Stripe.dev Blog

---

Executive Summary & Core Thesis

Developers often hold a flawed mental model that AI agents behave like diligent junior engineers who read documentation, browse project dependencies, notice deprecation warnings, and ask clarifying questions when stuck.

Stripe ran roughly a dozen steering experiments to understand how to make AI agents adopt current API versions, follow integration best practices, and leverage official agent skills. The results demonstrated that passive hints consistently fail, while active prompts and distribution plays succeed. Agents do not wander or explore; they operate like “contractors on a deadline” optimizing strictly for immediate task completion.

---

Taxonomy of Steering Experiments

1. Passive hints (Failed):
   • Warning hashes (warn objects) in API JSON responses.
   • Steering cues, READMEs, inline comments, and AGENTS.md files placed in dependency package roots.
2. Active prompts (Succeeded):
   • Progressive disclosure in skill files (modular on-demand sub-skills vs monolithic files).
   • CLI help output optimizations and explicit blocking errors in API compatibility mode.
3. Distribution plays (Succeeded):
   • Skill promotion on stripe login confirmation screens.
   • Skill install buttons and direct paths on docs.stripe.com.

---

Named Frameworks & Patterns (Verbatim)

• “Hard steering vs soft steering”: The foundational design axis for agent infrastructure. Hard steers (blocking errors, required context in prompt) enforce compliance; soft steers (warnings, hints, adjacent files) are ignored as noise.
• “Progressive disclosure in skill files”: Structuring agent skill definitions as modular, specialized sub-skills loaded on-demand rather than monolithic files.
• “Contractor on a deadline”: The operational model of agent behavior—agents do not browse, explore, or seek comprehension; they execute the immediate path to task closure.
• “Distribution problem, not a content problem”: Agent-facing developer experience succeeds or fails on context delivery (awareness → install → load → follow), not documentation prose quality.
• “Lost in the middle”: Attention decay in LLMs over long monolithic context files where middle tokens suffer degraded retrieval.

---

Quantitative Claims & Qualifiers

• Progressive disclosure advantage: Modular skill files outperformed monolithic blobs by roughly 10% across Stripe’s eval suite while meaningfully reducing token consumption.
• CLI login conversion rate: Embedding a one-click npx skill install prompt on the CLI login confirmation page yielded a 30–35% conversion rate.
• Passive hint efficacy: 0% effective response—agents completely ignored warn hashes in API responses and AGENTS.md in dependency folders.

---

Decision Rules

• When designing developer interfaces and APIs for AI agents, do use hard steers (blocking errors, explicit loaded context) rather than soft steers (warnings, hints, documentation files), because agents treat non-blocking warnings as noise and optimize strictly for task completion.
• When authoring agent skills and tool instructions, do use progressive disclosure (modular sub-skills), because it improves benchmark performance by ~10%, reduces token consumption, and prevents “lost in the middle” attention decay across long contexts.
• When steering agents toward current API versions, do fail hard with explicit blocking compatibility errors, because agents reliably detect and correct blocking errors but ignore soft deprecation warnings in response headers.
• When distributing agent skills to developers, do place actionable install prompts at high-intent CLI checkpoints (e.g., login confirmation screens), because conversion rates can reach 30–35% when presented at moments of high intent.

---

Anti-Patterns & Failure Modes (with Mechanisms)

1. SDK / Dependency Guidance (AGENTS.md in Packages)

• Mechanism: Embedding AGENTS.md or comments inside SDK dependency roots under the assumption that agents inspect project dependencies.
• Observable Symptom: Agents never navigate into node_modules or library source trees during task execution; the guidance remains completely unread and ignored.

2. API Response Soft Warnings (warn Hashes)

• Mechanism: Returning structured warning objects or deprecation notices alongside successful HTTP 200 responses.
• Observable Symptom: The agent extracts the specific data fields required for its immediate goal and discards the warning payload entirely, perpetuating deprecated API usage.

3. Monolithic Skill Files

• Mechanism: Placing exhaustive platform documentation and all API domain guides into a single large skill file.
• Observable Symptom: The LLM suffers attention decay (“lost in the middle”), wastes context window budget, and exhibits a 10% lower success rate compared to modular architectures.

---

What the Source Does NOT Claim

• Does NOT claim that high-quality human documentation is unnecessary; rather, human-facing DX is forgiving while agent-facing DX requires strict active context injection.
• Does NOT claim that agents possess intrinsic motivation to refactor or modernize codebases unless explicitly blocked by runtime errors or contractual prompts.