AI made code cheap. It didn't make judgment cheap

Tim Schumacher recently shared a story that stuck with me. A company in the portfolio adopted AI coding tools and went from about 5 to about 45 pull requests a week. Nine times the output. Revenue didn't move. The market simply didn't need nine times more features.

I don't think it's an outlier. Since 2022 our studio has delivered 97 projects, and I see the same pattern among the founders I work with: building got radically cheaper, and the results didn't follow.

The pattern

• The backlog grows instead of getting better. When anything can be built in a week, every request looks reasonable. The list never gets shorter – only longer and noisier.
• Teams track the numbers AI pushes up. Pull requests, story points, velocity – they climb on their own once agents write the code. They look great on a dashboard and say nothing about customers.
• Nobody owns "should we even build this?" Product assumes engineering will push back on weak ideas; engineering assumes product has already validated them. The question falls between the two.

Cost used to be the filter

When code was expensive, cost did a lot of quiet work. A feature that took three months had to be justified: someone wrote a business case, someone else argued against it, and half the ideas died in that conversation. It was a crude filter, but it worked.

Now the same feature takes a week, and the conversation gets skipped because it feels more expensive than the build. Bad ideas go through just as fast as good ones – and there are more of them, because the bar to propose one has dropped too.

And one cost didn't go anywhere. Writing a feature got cheaper; owning it didn't. Every new screen needs tests, support and documentation, and takes up space in the user's head. Every new integration is one more thing that can break at 3 a.m. AI-written code still has to be read, reviewed and maintained by people, and the product gets harder to use with every option nobody asked for.

Code got cheaper to write. It didn't get cheaper to own.

The new bottleneck is judgment

So the constraint is no longer engineering capacity. It's judgment – technical and product, usually both at once:

• What to build. Which problem is real, for whom, and which number should change once it's solved.
• What to kill. Features nobody uses, experiments that didn't pay off, integrations kept alive out of habit. Removing them is real work, and it rarely makes it onto a roadmap.
• What should stay boring and stable. Billing, authentication, data pipelines, the core flow customers pay for. That's where cheap speed is most dangerous: a quick change there can cost more than it will ever save.

None of this got faster with AI. If anything, it got harder, because the number of things waiting for a decision went up ninefold.

Measure outcomes, not output

Pull requests and velocity are inputs. They were weak proxies even when code was expensive; now that AI inflates them, they're actively misleading. A team that doubles its pull requests and moves no business metric didn't get twice as productive – it got twice as busy.

Better questions for a monthly review:

• Which metric did last month's releases move, and by how much?
• How many features shipped last quarter are still in use 30 days later?
• How long does it take us to remove something that didn't work?
• What share of the roadmap is tied to a clear hypothesis rather than a single request?

The same logic applies to AI projects themselves. We pick processes where a model can return its cost in 6–12 months, and agree on the target number before the build starts – more on that in where AI actually pays back.

Five questions before anything gets built

None of this needs a committee. These questions take ten minutes and replace the filter that cost used to provide:

1. What problem does this solve, and for whom? How do we know the problem exists – beyond someone asking for it once?
2. Which number should change, and by how much? If nobody can name one, it's not a feature yet – it's a guess.
3. How will we know in 30 days whether it worked? And if it didn't, who decides to remove it?
4. What does it cost to own for a year? Support, tests, infrastructure, and the extra complexity for users.
5. What are we not doing instead? The team's attention is still finite, even when its output isn't.

The point isn't to slow down. It's to spend the speed on things that matter – and to make killing a feature as normal as shipping one.

What this changes in the CTO's job

This is now the core of my work as a fractional CTO. A few years ago most questions were "how do we build it faster?" Today they're far more often "does it make sense to build this in the first place, and how will we know?" The job moved from managing engineering capacity to deciding what that capacity gets pointed at.

In practice it means sitting between product and engineering and owning the question nobody else owns: agreeing on success criteria before work starts, keeping the core of the system boring on purpose, and running regular reviews where features get removed, not just added. More about the format in our note on the fractional CTO role.

One question to take back to the team

What was the last feature your team shipped that no one asked for? If the answer comes quickly, the bottleneck has already moved – and it's worth deciding on purpose who owns it

Case · EdTech · LLM

Heritage School

Measured in hours, not features: 3–4 hours a week back for every teacher, and the timetable in 20 minutes instead of 2 weeks