Agentic Manual Testing

Automated unit tests are necessary, but code that passes unit tests can still fail to boot, miss styling, or crash on live input. Agentic manual testing is the discipline of having the agent spin up the software and actively interact with it before declaring completion.

Mechanisms

• CLI / One-liners: Direct the agent to execute edge cases directly via python -c "..." or compile temporary test scripts in /tmp.
• API Exploration: Have the agent start a local development server (e.g. python -m http.server or app backend) and explore JSON endpoints with curl.
• Headless Browser Automation: Direct agents using browser tools (Playwright, Puppeteer, agent-browser, or CLI wrappers like rodney) to click buttons, fill forms, verify layouts, and check console errors.
• Evidence Capture: Require verifiable outputs (terminal logs, rendered HTML, or tool execution transcripts) rather than accepting self-reported completion.

“First Run The Tests”

Starting a coding session with a prompt like "First run the tests" or "Run uv run pytest" forces the agent to discover test commands, gauges codebase complexity, and anchors the agent in a verification mindset from turn one.

Related

• simon willison
• agentic engineering patterns
• red green tdd
• tdd with agents
• karpathy four guidelines
• closed loop agent improvement
• automated eval engineering