LLM App Improvement Ladder

When an LLM app underperforms, don’t jump to fine-tuning. Pocock’s ordered ladder — try the simple thing first — moves from cheap prompt-level fixes to expensive architectural ones:^[raw/aihero/how-to-improve-your-llm-powered-app.md]

The Rungs

1. First prompt — the baseline; most failures are under-specified instructions.
2. Role-based prompting — give the model a persona/role.
3. XML tags — structure the input so sections are unambiguous.
4. Structured outputs — schema-constrain the response (structured outputs).
5. Reasoning — elicit step-by-step thinking (cf. think tool).
6. Multishot prompting — examples in-prompt.
7. Temperature — tune randomness to the task.
8. Tool calling — let the model act (tool calling loop).
9. LLM call chaining — decompose into a pipeline (workflow pattern, agents vs workflows).
10. RAG — retrieve documents into context.
11. Chunking — tune retrieval granularity.
12. Agentic loops — iterate tools against environment feedback.
13. Parallelizing LLM calls — fan out independent work.
14. Evaluator-optimizer — generate/evaluate loop (generator evaluator loop).
15. LLM routers — classify inputs to specialized handlers.
16. Fine-tuning — the last rung, not the first.

Why the Order Matters

Each rung’s cost compounds: prompt fixes are free, structural patterns cost engineering time, fine-tuning costs data plus pipeline maintenance. The ladder is also the diagnostic sequence — an app that fails at rung 3 won’t be saved by rung 16. Measured by eval taxonomy at every step, or you’re climbing blind.

Related

eval taxonomy, structured outputs, tool calling loop, agents vs workflows, generator evaluator loop, smart zone.