Engineering Leadership,  Applied AI

Validation Is the New Bottleneck: Reviewing AI Output You Can't Read Line-by-Line

By Isaac Martin
Hero image for the post 'Validation Is the New Bottleneck: Reviewing AI Output You Can't Read Line-by-Line'

Run one coding agent and you can still review its work the old way: open the pull request, read the diff, leave comments. Run three or four in parallel and the math stops working. The agents produce correct-looking changes faster than any human can read them carefully, and the scarce resource quietly shifts from building to specifying and validating.

I want to be precise about what I'm claiming here. This isn't a forecast about some future where machines write inscrutable code. This is a present-tense process problem for teams already running agents in parallel — and I think most of us are handling it badly, by pretending the old review ritual still works.

The assumption underneath the pull request

The traditional PR ritual rests on an assumption so old we forgot it was an assumption: a human can and should read every line before it merges. That assumption held when a senior engineer produced a few hundred lines of considered change per day. It does not hold when a set of agents produces several thousand lines an hour across parallel workstreams.

What happens when teams cling to it anyway is predictable. Review becomes the bottleneck, so reviewers start skimming. The ritual survives — approvals get clicked, the process looks intact — but the substance is gone. That's the worst of both worlds: you've kept the cost of the ceremony and lost the assurance it was supposed to buy. A rubber stamp is more dangerous than no stamp, because it manufactures false confidence.

The honest move is to admit the assumption broke and redesign the process deliberately, rather than let it degrade by attrition.

Spend review attention on boundaries, not statements

If you can't read every line, the question becomes: where does a unit of human attention buy the most assurance? In my experience, it's at the boundaries, not in the implementation:

  • Contracts and interfaces. Did the change alter an API shape, a database schema, an event payload, a public type? These are the surfaces where a defect propagates. I read every line of a migration or an interface change; I skim the internals of a function that satisfies its tests.
  • Tests as the reviewable artifact. I've largely stopped asking "is this implementation right?" and started asking "do these tests pin down the behavior I actually specified?" A test suite is short enough to read carefully, and it's the thing that keeps being true after the next agent rewrites the internals.
  • Invariants. The properties that must never break — authorization checks, money math, idempotency, data ownership — deserve explicit assertions and explicit review. Name them, encode them, and check that the change respects them.
  • Telemetry. Some validation belongs after the merge, not before it. Error rates, latency, and business metrics on the changed path will catch a class of defects no diff reading ever would. Budget for watching the change land, not just approving it.

The common thread: validate behavior and boundaries, not every statement. This is how validation cost scales sublinearly with output volume — the number of contracts and invariants grows far slower than the number of lines.

The real leverage is upstream

Here's the part that took me longest to internalize: at agent velocity, the quality of what you hand the agent — the spec, and just as much the acceptance criteria — determines output quality more than any downstream review does. Ambiguity in, defects out.

The spec tells the agent what to build; the acceptance criteria tell it — and you — how to know it's built. When I write acceptance criteria carefully — concrete inputs and outputs, named edge cases, explicit non-goals — the agents converge on something reviewable. When I hand over a vague ticket, I get plausible code that solves a slightly different problem, and no amount of diff reading reliably catches "solved the wrong problem." I've written before about making AI tools plan before they code; this is the same principle applied to the team process. Specification is the new senior work. Review is increasingly just checking that the output matches criteria good enough to check against.

Practically, our engineering team at PlantBid has found that the spec and the test plan are worth more review discussion than the diff itself. Argue about the acceptance criteria before the agent runs, and the PR review becomes verification rather than discovery.

Where you should still read every line

I want to be honest about the trade-off, because this approach raises real risk in specific places. Behavioral validation depends on your ability to specify behavior and test it. Some code resists both:

  • Security-sensitive paths: authentication, authorization, cryptography, anything touching secrets.
  • Code that's genuinely hard to test: concurrency, complex failure handling, subtle performance characteristics.
  • Anything with a large blast radius and a slow recovery: migrations you can't roll back, billing, data deletion.

For those, keep line-by-line review. The point isn't that careful reading is obsolete — it's that "read every line" should be a deliberate risk/reward decision, not an unexamined default. An internal tool used by three people carries a very different consequence for a missed defect than a billing path does; spending identical review attention on both is how reviewers burn out and start skimming everything, including the code that mattered.

And be clear-eyed about what you're trading when you skip the careful read. Letting an agent operate as a software engineer without that tight control will let problems creep into the system. That's technical debt, like any other — and like any other debt, you should take it on intentionally and get something very real in return that justifies it. Velocity on a low-stakes internal tool might justify it. Velocity on your authorization layer does not.

A note on timing

I'll add one caveat: this problem is real but young. The level of parallel-agent throughput that breaks per-line review has only recently become practical — for me, it arrived with Fable-style parallel agent orchestration, and even now I'd say we're on the cusp of it being viable rather than comfortably past it. If your team runs one agent at a time, your existing review process probably still holds. But the throughput curve only points one direction, and the teams that redesign validation deliberately will be in a much better position than the ones that let their review culture quietly rot into rubber-stamping.

The bottom line

When output stops being scarce, validation becomes the bottleneck — and a validation strategy that scales linearly with lines of code is a strategy that fails. Move your attention to contracts, tests, invariants, and telemetry. Move your rigor upstream into the spec and the acceptance criteria. Reserve line-by-line reading for the code where the risk justifies the cost — and treat any review you skip as debt you've chosen deliberately, not a corner you've cut.

Worth asking of your own team: of the approvals you clicked last week, how many involved actually reading the code — and would you rather admit that in your process design, or keep pretending?