C3. From Pilot to Production
Most AI pilots never reach production. They get stuck at the demonstration stage: successfully shown off technically but never carried into regular operations (so-called pilot purgatory).
What this chapter delivers: the approach for building a pilot for the transition from the very start, the scaling criteria and the go/no-go checklist for the Pilot → Scale gate from chapter C1, and the rollout strategies for the path to broad adoption.
Setting up the pilot right from the start
Whether a pilot reaches production is mostly decided before it starts, with the question of what it is built as. Two mindsets stand opposed; both describe the same pilot, just set up differently:
- Pilot in demo mode: built to convince in a demonstration. Everything is optimized for that date: selected data, its own environment, no reviews. After the demonstration, none of it is usable in production, and the real work starts over. This is exactly how the pilot purgatory from the chapter introduction comes about.
- Pilot as a production precursor: built as a small edition of the later production system: the same data, the same platform, the same reviews, just at small scale. If the gate decision is go, this pilot grows into production instead of being rebuilt.
Every row of the table is a design decision made when the pilot is set up. The last column shows why the decision makes the difference: what happens at the transition to production.
| Design decision | Pilot in demo mode | Pilot as a production precursor | Consequence at the transition to production |
|---|---|---|---|
| Success criterion | The pilot starts with no fixed goal, in the spirit of “let’s see what’s possible”. | A measurable criterion and a baseline are fixed before the start (see chapter D1). This way the pilot proves or disproves a concrete hypothesis. | The gate needs evidence: the demo pilot delivers only impressions, the production precursor delivers measurements against the baseline. |
| Data | The pilot works with hand-picked examples and a pre-cleaned data extract. | The pilot uses real production data with all its messiness, via the real access path. | The demo pilot’s quality collapses as soon as real data arrives; the production precursor has already cleared this hurdle in the pilot. |
| Architecture | The pilot runs in an isolated test environment (sandbox) that cannot be integrated later. | The pilot is a lean version of the target picture (see chapter B5): the same platform building blocks, the same authentication, just smaller. | The demo pilot has to be rebuilt for production; the production precursor is extended instead of replaced. |
| Costs | Costs are not measured, because “it’s only a pilot”. | Cost per unit (request, transaction) is measured from day 1 and extrapolated to the target scope (see chapter D2). | Whether ongoing operation pays off shows up only after rollout for the demo pilot, but already at the gate for the production precursor. |
| Governance | Security, data protection, and ethics are deferred to “later”. | The reviews run within the pilot (see chapters B8 to B10). Approvability is part of the pilot result. | The catch-up review blocks the demo pilot’s rollout for months; the production precursor brings the approvals with it. |
| Users | The project team tests itself. | Real users from the business unit work with it close to production and give structured feedback. | Acceptance and everyday fitness show up only at scale for the demo pilot; the production precursor knows both from the pilot already. |
| Timeframe | The pilot runs without a fixed end date. | The pilot has a time box of 3 to 6 months with a fixed gate date. | Without an end date the demo pilot becomes a permanent makeshift; the time box forces the gate decision. |
The extra effort of this discipline is small compared to its effect: a pilot set up this way answers all questions of the go/no-go checklist at the gate from its ongoing operation, instead of having to reconstruct them afterwards.
Why pilots fail
The table shows the most common causes of failure with their typical symptom. It serves two purposes: as an early-warning list during a running pilot (is one of the symptoms already showing?) and as a root-cause analysis for already failed efforts, before the next pilot starts.
| Cause | Symptom |
|---|---|
| No success criterion defined | The pilot “works,” but nobody can say whether it achieved the expected benefit |
| Tested on demo data instead of production data | Quality collapses once real, messy data is processed |
| No architecture connection | The pilot runs in isolation and can’t be integrated into core systems without a rebuild |
| Inference costs underestimated | The pilot was cheap because it handled few requests; ongoing operation doesn’t pay off |
| No owner after the pilot ends | The project team disbands, nobody owns the transition into operations |
| Missing governance approval | Security, compliance, or ethics review was never planned for the pilot and now blocks the rollout |
| Too many parallel pilots | Capacity is spread across too many efforts, none of them carried through consistently |
Scaling criteria
A pilot is ready to scale only when it satisfies all three dimensions, not just the technical one. The three lists below check technology, organization, and economics; each item is backed with evidence at the gate and feeds into the go/no-go checklist in the next section.
Technical
- The solution was tested with real production data, not just curated examples
- The architecture connects to core systems without a rebuild
- Observability (metrics, logs, drift detection) is in place
- The security checklist has been fully completed (see chapter B9)
Organizational
- A service owner for ongoing operations has been named (see chapter B7)
- Governance, compliance, and ethics approvals are in place (see chapters B8 and B10)
- A support structure (tiers 1 to 3) is defined
- Affected business units are informed and involved (see chapter B11)
Economic
- The business case was updated with actual pilot data, not just the original estimate
- Ongoing costs (inference, operations, licensing) are extrapolated for the target scope
- The pilot-phase success criterion was demonstrably met, or adjusted with justification
- Cost allocation for ongoing operations is clarified
Template: go/no-go checklist
How to use the checklist: before the Pilot → Scale gate decision, every row is filled in with concrete evidence. The last column fixes in advance what happens if a criterion is not met. That is the decisive difference from an ordinary list: the consequence is not negotiated in the meeting, when the pressure to keep going is at its highest.
| Criterion | Met? | Evidence | Decision if not met |
|---|---|---|---|
| Success criterion achieved | Extend the pilot with an adjusted goal, or end it | ||
| Tested with production data | Don’t scale; retest with real data first | ||
| Architecture connectable | Reassess integration effort, adjust architecture if needed | ||
| Governance approvals in place | Block scaling until approval is granted | ||
| Service owner named | No handover into operations without a named owner | ||
| Economics confirmed at target scope | Revise the cost model or reduce scope |
Decision rule: a “no-go” on technical or governance criteria is binding. Here you fix the issue, not bypass it. On economic criteria, management can deliberately and transparently deviate if strategic reasons outweigh it.
Rollout strategies: the path to broad adoption
After the go, the question is how the solution reaches the whole footprint. Four strategies are established; they can be combined (e.g., shadow mode per new user group within a staged rollout).
| Strategy | Approach | When suitable | Limits |
|---|---|---|---|
| Shadow mode | The AI runs in parallel with the existing process; its results are compared but not used | Consequential decisions (credit, pricing); whenever quality has to be proven on real volume first | Double effort for the duration; no end-user feedback because nobody works with the results |
| Staged rollout | Expansion by unit, location, or user group with a checkpoint per stage | The standard case for most applications; each stage delivers lessons for the next | Requires patience and disciplined follow-through on the checkpoints |
| Parallel run with control group | Part of the users or transactions work with AI, part without; results are compared | When the proof of impact (against the baseline) has to be robust, e.g., for the business case | Organizationally demanding; weakly informative at small case volumes |
| Big bang | All users at once | Only for low risk, proven technology, and easy rollback (e.g., an internal assistance tool) | If problems occur, everyone is affected at once; unsuitable for consequential or customer-facing systems |
For every strategy: a way back is defined before the rollout (shutdown path, fallback to the old process). A rollout without a defined way back is a bet, not a plan.
After the go: transition and stabilization
Industrialize instead of scaling up. The pilot infrastructure is replaced by the production-ready platform, not simply scaled up as-is: the evaluation suite goes into the deployment pipeline, monitoring and alerting follow the operations standard, rate limiting and budget alerts are active, access and logging follow the security checklist. Cost per unit is measured again at the target volume: economies of scale and load peaks shift the pilot figures.
Orderly handover. The project team hands over to the service owner with full documentation: model card, architecture decisions, known limits and open risks, a runbook for operations and incidents, business-side contacts. Success criteria are carried into ongoing reporting (see chapters D1 and D4).
Plan a hypercare phase. The first 4 to 6 weeks after the rollout are a defined stabilization phase with intensified support: the development team remains partially available, support paths are shortened, quality and cost metrics are watched closely, user feedback is actively collected instead of waited for. The hypercare phase ends with a defined criterion (e.g., two weeks stable within the service targets), not with a date alone; after that, regular operations apply (Operate phase, see chapter C1).
Checklist: from pilot to production
The checklist accompanies the entire path of this chapter, from pilot design to the hypercare phase:
- Pilot set up as a production precursor (design table above), not as a demo
- Success criterion and baseline fixed before the pilot start
- Root-cause analysis of previously failed pilots evaluated
- Scaling criteria checked across all three dimensions
- Go/no-go checklist completed and decision documented
- Rollout strategy chosen and justified; way back defined
- Service owner and support structure named before rollout
- Pilot infrastructure transferred to the production-ready platform; cost per unit re-measured at target volume
- Handover documentation complete (model card, runbook, open risks)
- Hypercare phase planned with an exit criterion
- Success criteria carried into ongoing reporting