Agentic Code Quality

By Addy Osmani (originally published on Substack).

For much of human history, we’ve evaluated code quality via code review: someone reads what you wrote and makes sure it’s clean, thoughtful, fast, understandable, and tests well. For agents, that approach doesn’t scale well; there’s just too much code for anyone to read. As a result, more and more of our quality checks have to happen in the harness, environment, and operating system around the agent. I still read and review code, but am very intentional about where I am comfortable with constraints as the check.

Software quality now depends on the constraints you set around your agents.

Constraints define what the system is allowed to do by throwing tests and deterministic constraints at an agent’s proposals. It’s by setting and maintaining these constraints that we build loops that reliably deliver high-quality production software, even when agents are creating hundreds of thousands or millions of changes every single day.

We call these constraints quality gates, and they take many forms. They include conventional unit tests, property tests, and acceptance tests. They include mutation testing, where we generate variations of code, run it against the same tests, and make sure that people aren’t sneaking bugs in that we’re missing. They’re metrics around code quality, such as cyclomatic complexity and line length, that help keep things readable. Constraints also play an important role in what proposals the system will accept and apply as code changes. By the time a change proposal moves from the interpreter running the agent to the agent controller and out to production, we’ve done enough checks on it to be confident that it’s safe to ship and the impact of its change is well within the scope of the agent.

Guillermo Rauch’s list is a good test for whether you can afford to skip reading: low stakes situations (no users, throwaway code, prototype). Once the stakes go up, something has to read the code. If it isn’t you on every diff then it has to be the constraints.

An agent can propose anything. Your constraints decide whether a proposal is safe enough, correct, scoped, and useful, for you and your team to ship.

This model offers a lot, but also leaves out many pieces:

Autonomy: agents might apply intentions well, but fail when there is missing info or ambiguity.
Trust: we start with trust, but it has to be hard-earned based on risk, evidence, and track record.

Some constraints shape work before it begins. Others give feedback while the agent is working. Others decide whether its output can cross the production boundary at all.

In my experience it helps to have a broader set of checks instead of solely relying on unit tests: type safety, performance, late-stage security scanning, and architecture rules (e.g. ESLint / boundaries).

For now, much of the difference between useful agent output and slop still comes down to the skill of the team operating the loop.

Human attention is scarce and valuable so we should proactively direct it to those most nuanced problems that require our judgment. Downstream humans should only be pulled in when automated guardrails break.

Back-pressure can be implemented through many tools: compilers rejecting invalid code, tests failing, security policies blocking bad practices, CI declining to deploy. Ideally it exists throughout the loop. If we run out of room in the verification loop:

Scale the verification system and create more capacity.
Reduce the rate at which agents generate changes.
Shift/adjust quality bars intentionally.

Quality is in the constraints that we place around our agents.