Why AI pilots stall — and five things that get them to production

● In short

Run the pilot next to the current process and compare metrics, not impressions.
Integrate where people already work — the CRM, the messenger, the existing form.
Production means monitoring, rollback and a named owner on the client's side.

Our process has four phases: hypothesis, prototype, pilot and production. The first three are where most of the visible work happens, and where most of the excitement is. The fourth is where projects quietly die. Over the years we have collected a short list of things that make the difference. None of them is about the model.

1. The pilot runs next to the current process

A pilot that replaces the current way of working from day one has no baseline. When something goes wrong — and something always does — there is nothing to compare with, and the discussion turns into impressions: “it feels slower”, “the managers don't trust it”. We run pilots on part of the flow, alongside the existing process, for weeks rather than days. Then we compare the numbers agreed during the hypothesis phase: time per item, error rate, conversion, cost.

2. The model lives where people already work

Every new screen is a reason not to use the system. In RED, a real estate business, the AI agent qualifies inquiries from WhatsApp, Instagram and Telegram and hands a ready-to-work lead — property details, budget, timeline — straight into the CRM the team already used. Nobody had to learn a new tool; their pipeline just started to arrive pre-filled. That is a big part of why it freed around a fifth of the team's working time instead of becoming one more tab.

If people have to go somewhere new to use the model, most of them won't.

3. Quality is monitored after launch, not just measured before it

A model that scored well on a held-out sample in week four will drift. Customers start asking different questions, a new product line appears, a form changes upstream. Production needs ongoing quality monitoring: a sample of outputs reviewed every week, alerts when confidence drops or the share of “I don't know” answers grows, and a simple way for users to flag a bad result. In Quidly, where an ML model scores microloan applications and matches them to the lenders most likely to approve, watching the approval rate of the model's recommendations over time is as important as the model itself.

4. There is a rollback

Every production AI system we hand over has a documented way to switch the model off and fall back to the manual process — in minutes, by the client's own team. It is rarely used. But knowing it exists changes how willing people are to rely on the system, and how quickly management agrees to expand it.

5. Someone on the client's side owns it

The most reliable predictor of whether a system is still running a year later is whether a named person in the client's team is responsible for it — not for the code, but for the outcome. They read the weekly quality sample, decide when to retrain, and are the first person users go to. During the pilot, we work with that person every week; by handover, they know the system better than we do.

What a handover looks like

By the end of the production phase the client's team should hold everything needed to run the system without us:

The code, prompts and model configuration in the client's own repository and cloud account.
A runbook: how to deploy, how to switch the model off, who to call when something looks wrong.
The evaluation set used in the pilot, so any future change — a new model version, a new prompt — can be checked against the same examples.
The quality dashboard and the weekly review routine, already running for a few weeks with their owner in charge.
A short list of known limitations: the cases where the model is weak and a human should step in.

Handing this over feels like making ourselves unnecessary. That's the point. A system the client can't run alone isn't in production; it's a dependency.

A quick self-check

If your AI project has been “almost ready” for a while, ask these questions:

Is there a number, agreed in advance, that the pilot has already beaten?
Do users reach the model from a tool they already use every day?
Who looks at the model's output quality each week — and what do they do if it drops?
How long would it take to switch the model off tomorrow?
Whose name is next to the system in the org chart?

Two or more blank answers usually explain the stall. The good news: none of these requires rebuilding the model. They are a few weeks of integration, monitoring and handover work — the part of the project that turns a pilot into the way the work is done.

Why AI pilots stall — and how to get them to production