← All notes
Note 09Engineering

How do you know AI actually pays off in engineering?

Every board meeting now asks for AI ROI. The usual answers – token cost and "developers feel faster" – don't tell the CEO much. Here is what I would measure instead, and when to start.

● In short

  • Token cost is a cost, not an output. And the feeling of speed rarely survives more than one quarter.
  • Measure the delivery process instead: lead time, share of planned work, failure rate, cost per delivered feature and where the bottleneck moved.
  • Take the baseline before rollout, not after – otherwise you're comparing to memory.

So how do you know AI actually pays off in engineering? Every board meeting now asks for AI ROI, and the typical answers I get are "token cost" or "developers feel faster". Neither tells the CEO much.

Both answers describe the tool. The board is asking about the business: are customers getting what they need sooner, and is it costing less to get it to them? That's a question about the whole way work moves from a request to production – and it needs different numbers.

Why the usual answers don't work

Token cost is a cost, not an output. Knowing the team spent a certain amount on models last month says nothing about what the business got for it. It's half of the ROI formula presented as the whole thing.

"Feeling faster" is real, and I don't dismiss it. The first version of a function, a test or a migration really does take less time to write. But that feeling rarely survives more than one quarter. The time saved on typing quietly moves somewhere else: longer reviews, more rework, more conversations about what just got shipped. The developer feels faster. The company doesn't see it.

It's the same trap as counting pull requests: AI inflates output metrics on its own, so they stop meaning anything – more on that in AI made code cheap, not judgment. The numbers worth tracking describe the process, not the tool.

1. Lead time from request to production

Not lines of code or pull request count, but how long it takes for a change the business asked for to reach the customer. Start the clock when the request is accepted and stop it when the change is live.

This is the metric the CEO actually cares about, even if they don't call it that. If lead time doesn't move after AI tools are rolled out, the speed-up happened somewhere that doesn't matter to the business. Look at the median and at the slowest requests separately: an average hides the changes that sit in a queue for a month.

2. Share of time spent on planned work

Coding faster does nothing if the team spends even more time fixing things it has already released. Look at how time splits between planned work – the roadmap – and unplanned work: bugs, incidents, urgent fixes, rework after review.

If output goes up but the share of planned work goes down, the team is running faster to stay in place. Getting this number doesn't require a new system. Two labels in the task tracker, planned and unplanned, checked once a sprint, are enough to start.

3. Failure rate

If generated code passes review and then breaks in production, that's negative ROI. You paid for the generation, the review, the incident and the fix – and the customer saw the failure.

Count the share of releases that end in a rollback, a hotfix or an incident, and how long recovery takes. Watch it most closely in the first months after rollout. Generated code tends to look plausible, and reviewers get used to approving plausible code faster than they should.

4. Cost per delivered feature

Add up tokens, tools and review time, then divide by the number of features actually shipped to customers. Shipped means released and in use, not merged.

Review time is the part most often left out, and with AI it grows: someone has to read every generated change carefully, and senior engineers' hours are the most expensive line in the budget. If tokens are cheap but every feature now takes twice as long to review, the cost per feature may have gone up, not down.

This is the number that belongs on the board slide next to token spend – it's what gives the spend a meaning.

5. Shifting the bottleneck

As coding gets faster, the backlog moves to QA, review or product decisions. Code waits for testers, pull requests wait for the two people allowed to approve them, finished features wait for someone to decide whether to release them.

Solve the new bottleneck, or the gains are gone. In practice that means looking at where work waits, not where it gets done: time in review, time in testing, time between "ready" and "released". Quite often the next investment isn't another AI tool but automated tests, clear ownership of review or faster product decisions.

Measure the baseline before rollout

The most common mistake isn't picking the wrong metric. It's starting to measure after the tools are already in place. Then "before" is whatever people remember – and memory is generous to the new thing.

A baseline doesn't have to be perfect. A couple of months of history from the task tracker and the release log is enough: lead time, the split between planned and unplanned work, failed releases, time in review. Write the numbers down before rollout and compare the same numbers a quarter later.

It's the rule we follow on every AI project: agree on the target number and measure the starting point before the build, not after. More in where AI actually pays back.

Measure the process, not the tool

Using AI in engineering isn't buying a tool. It's changing the way you work.

The license is the easy part. The real change is in how the team plans, reviews, tests and releases – and that's where the return either shows up or disappears. A tool can be bought in a day. A process takes a quarter to change and another quarter to measure.

If you need to answer the board's question about AI ROI and the numbers aren't there yet, setting up this measurement is part of my work as a fractional CTO.

A question for your team

How does your team measure this today? If the honest answer is "by feel", the baseline is the first thing to fix – before the next tool gets rolled out

Case · EdTech · LLM

Folio

Lead time that a business can see: an adaptive AI editor for a large corporate university, built and launched in six weeks

● More notes

Is it just a wrapper? The five questions behind the question

Raise money for an AI startup and you'll hear "is it just a wrapper?" It's really five questions: moat, data ownership, unit cost at 10× traffic, swapping the model in a week, and proof that it works.

Fundraising6 min read

AI made code cheap. It didn't make judgment cheap

AI tools took one team from 5 to 45 pull requests a week while revenue stayed flat. When code is cheap, the bottleneck moves to judgment: what to build, what to kill and how you'll know it worked.

Product5 min read

Contact

The board wants AI ROI? Measure the process

Tell us how the team works today and which AI tools are in place or planned. We'll help set the baseline and pick the numbers that show whether it pays off

Start a conversation →