Planning an AI Pilot: Design, Scope and Success Criteria
Plenty of AI initiatives start as a "pilot" and never finish. The system runs somewhere on the side, nobody measures anything that would stand up in a budget meeting, and months later no one can say whether it was worth it. This guide shows how to set up an AI pilot so that, after a fixed period, it produces a clear decision: scale it, adjust it or stop it. Current as of October 2026.
What is an AI pilot, and what is it for?
An AI pilot is a time-boxed deployment of an AI system in live operations, with real users and real cases, to test a hypothesis written down beforehand. It exists to support an investment decision, not to prove that AI "kind of works". Without an agreed decision at the end, it is a trial with no exit.
It helps to separate the pilot from the phases on either side of it:
- Feasibility prototype (proof of concept): tests with sample data whether a task can be solved at all. Output: it works, or it does not.
- Pilot: tests in live operations whether the solution delivers the expected value on real data, real exceptions and real people, and whether they actually use it. Output: a reasoned go, adjust or stop.
- Production: scales a solution whose value has been shown, with operations, monitoring, support and clear ownership.
The transition is where many projects stall. According to a Gartner forecast from 2024, as reported by Intelligent CIO, at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, escalating costs and unclear business value among the reasons. For agentic AI, Gartner said in June 2025, as reported by W.Media, that more than 40% of projects could be cancelled by the end of 2027. Both are forecasts, not measurements. They do point to the same gap: proving value, not building the technology.
What does a well-designed pilot consist of?
A sound pilot has six parts: a testable hypothesis, a measured baseline, a tight scope, a measurement plan, a fixed duration and decision thresholds agreed in advance. Leave any of them out and the results can be read whichever way suits the reader, which is exactly what prevents a clear decision.
Hypothesis
The hypothesis links an intervention to a measurable effect in a specific process. Weak: "AI will help sales." Usable: "If an AI agent drafts quotes for standard enquiries from the incoming email and the product master data, median time from enquiry to sent quote will be below today's figure, and the error rate found at review will not rise." It names the process, the intervention, the metric and a guardrail that must not get worse.
Baseline
Without a baseline there is nothing to compare against. Measure the target metric before launch, in the same slice of the process and using the same method you will use during the pilot. The data often already exists in your ticketing system, ERP or mail archive: timestamps for receipt and dispatch, number of follow-up questions, corrections at approval. Where nothing is recorded, have people log it by hand for two or three weeks.
Scope
Scope defines which cases, which users and which systems are in the pilot, and which are explicitly out. A narrow cut is not a lack of ambition. It is what makes clean measurement possible: one product line, one site, one type of request.
Measurement plan
The measurement plan states which metrics are collected, how often and from which source. Anthropic's Claude documentation recommends success criteria that are specific, measurable, achievable and relevant, and notes that most use cases need evaluation across several dimensions at once. For a pilot, that means measuring value, quality, adoption and cost separately.
Duration and decision thresholds
Duration and thresholds are fixed before launch and signed off by the decision-maker. There are usually three outcomes: go (thresholds met, plan the move to production), adjust (partial success, one clearly named bottleneck is addressed in a short second round) and stop (thresholds missed or a guardrail breached). "Let's watch it for a few more months" is not a valid outcome.
Which pilot formats are there?
Pilots differ mainly in how much responsibility the AI system already carries. Four formats cover most situations. The more responsibility the system takes on, the stricter the thresholds, fallback and monitoring need to be.
Shadow mode. The system processes real cases in parallel, but its output never leaves the building. Staff work as usual, and afterwards you compare human and AI results. This suits classification, data extraction and drafting where errors would be costly. It measures quality well, but tells you little about adoption or time saved.
Assisted mode with approval. The system drafts something, such as a quote, a ticket reply or an inspection report, and a person reviews and approves it. This is the most common format and usually the most useful, because it captures real working time, real corrections and real adoption without unreviewed output reaching customers. For quotes, this is the typical starting point for a quotation agent; in field service, for a technical service agent.
Partial autonomy. For clearly defined routine cases the system acts on its own, and everything else goes to a person. This needs reliable case routing and spot checks on the cases handled autonomously. It only makes sense once an assisted pilot has shown that quality on routine cases is stable.
A/B comparison. Two comparable groups work over the same period, one with AI support and one without. This gives the cleanest read on impact, but needs enough cases and enough people that chance and individual performance do not drive the result.
Whether the technology and the data hold up at all is a question for before the pilot. An AI feasibility assessment checks data, system access, error tolerance, model capability and organisation separately. The pilot starts where feasibility has been answered with a yes, and asks a different question: not "can it be done?" but "does it pay off in operations?". A short feasibility prototype can sit in between. What comes after is the move into production, with operations, monitoring and ownership. Both have their own questions and do not belong inside the pilot period.
What has to be in place before you start?
Three things need to be settled before launch: access to real data and systems, a business owner for the process, and clarity on data protection and employee consultation. If any of these is missing, the pilot spends its time on organisation instead of measurement.
Data and systems. Agree up front which systems the AI may read from and write to: shared mailbox, ERP, document store, ticketing. Read access is enough for many pilots; grant write access only where assisted mode needs it. Spot-check data quality as well. Missing product data or outdated price lists will make any agent look bad, and that is not an AI problem.
Business ownership. Name a person who knows the process, supports users during the pilot and has time for the evaluation. Separately, name the decision-maker who holds the budget. Neither is the same as the team building the system.
Employee consultation. If the system can record how individual staff work, for example logs of who approved how many drafts and how fast, employee representatives may need to be consulted, depending on the jurisdiction. In Germany, works councils have a co-determination right over technical systems capable of monitoring employee behaviour or performance under Section 87(1) no. 6 of the Works Constitution Act. Agree early that pilot data will not be used to assess individuals. This is orientation, not legal advice.
AI literacy. Since 2 February 2025, Article 4 of the EU AI Act has asked providers and deployers of AI systems to take measures on AI literacy for the people who operate them. For a pilot that means, in practice, a short and documented briefing for users on how the system works, where its limits are and what to do when it gets something wrong. A kick-off workshop often covers this along the way.
Data protection. Be clear about where the model runs and which data it sees. For sensitive data, consider models hosted in your own data centre or by a European provider; the background is covered under sovereign AI. A pilot on live data can be operated in line with the GDPR once the legal basis, processing agreements and retention periods are settled.
How do you plan a pilot step by step?
Plan in seven steps: pick the process, write the hypothesis, measure the baseline, cut the scope, fix metrics and thresholds, get the pilot charter signed, then build and launch. The first six steps take little time but decide whether you end up with a result anyone will trust.
- Pick the process. Good candidates are frequent, rule-based with exceptions, and text-heavy: quote requests, service requests, supplier documents, internal knowledge questions. Poor candidates are rare, politically contested or have no recorded data.
- Write the hypothesis. One intervention, one primary metric, one or two guardrails. Phrase it so an outsider could answer it with yes or no at the end.
- Measure the baseline. Same metric, same slice of the process, same method as in the pilot. Record the spread too. If turnaround currently ranges from one hour to three days, you need more cases to detect a difference.
- Cut the scope. Write down which case types, users and systems are in and which are out. Listing the exclusions explicitly stops scope from creeping during the pilot.
- Fix metrics and thresholds. For each dimension: one metric, one source, one threshold for go and one for stop. Also set the minimum number of cases before you evaluate.
- Get the charter signed. The decision-maker confirms hypothesis, scope, duration and thresholds in writing. This protects against goalposts moving later.
- Build, brief, launch. Only now does the system get built. Keep it lean: it has to cover the pilot scope reliably, not the eventual full rollout.
A pilot charter can be as short as the example below. The values are placeholders for your own measurements, not expected results:
pilot: quote-drafts-standard-requests
hypothesis: >
An AI agent drafts quotes for standard requests from the incoming
email and product master data; turnaround is below baseline and
the correction rate at approval does not rise.
scope:
in: [standard requests product line A, inside sales team north]
out: [custom builds, framework contracts, export customers]
duration: { start: 2026-11-09, end: 2026-12-18, min_cases: 150 }
baseline:
turnaround_median_h: <measured>
correction_rate: <measured>
metrics:
value: turnaround_median_h # source: ERP timestamps
quality: correction_rate # source: approval log
adoption: share_of_drafts_used # source: agent log
cost: cost_per_case # source: provider billing
thresholds:
go: "turnaround < baseline AND correction_rate <= baseline"
stop: "correction_rate > baseline OR adoption < 0.5"
decided_by: head of inside sales
Which success criteria work for an AI pilot?
Useful success criteria cover four dimensions: value in the process, quality of the output, adoption by users and running cost. Each dimension gets a metric with a source and a threshold. Measuring only one of them means deciding on half the picture.
| Dimension | Example metric | Typical source |
|---|---|---|
| Value | Turnaround time, cases per person per day | ERP or ticket timestamps |
| Quality | Correction rate, errors in a sample | Approval log, review |
| Adoption | Share of drafts used, user feedback | Agent log, short survey |
| Cost | Model and operating cost per case | Provider billing |
Two rules help. First, include guardrails. Faster handling is worthless if the error rate goes up. Second, qualitative impressions matter but do not replace numbers. Capture feedback in a structured way, for instance a 1 to 5 rating plus a free-text field, and review it alongside the metrics.
For the technical quality of the AI system itself, keep a small test set of real, anonymised cases with known correct answers. Run it before launch and after every change to the system. That way you can tell whether a tweak made during the pilot quietly made things worse.
What are the limits and risks?
A pilot can only answer questions about what happens inside its scope. Rare exceptions, peak loads and long-term effects often stay invisible. On top of that come common planning mistakes that distort the result. Knowing them lets you rule them out in the charter, or at least name them openly.
- Target without a baseline. Without one, every result becomes a matter of opinion.
- Thresholds moved after launch. Adjusting criteria mid-pilot means you end up measuring whatever looks good.
- Scope creep. "Could it also handle…" is the most common reason pilots never end.
- Showcase team instead of normal operations. A pilot staffed only with enthusiasts overstates adoption.
- Novelty effect. People experiment and praise more in the first days. Evaluate the early and late phases separately.
- Hidden costs. Model fees, integration effort and maintenance belong in the calculation, or a go decision will not hold up. The guide on how to calculate AI ROI shows how to turn pilot measurements into a full-lifetime business case; the pilot supplies measured inputs instead of estimates.
- No plan for afterwards. Even a successful pilot fails if nobody takes over operations, monitoring and further development.
Structured risk work does not have to be heavy. The voluntary NIST AI Risk Management Framework organises it into four functions: Govern, Map, Measure and Manage. For a pilot it is often enough to answer four matching questions in the charter: who owns what, where does the system act, how is it measured, and what happens when it gets something wrong? How these questions are settled permanently across the organisation is covered in the overview of an AI governance framework.
Finally, not every process needs an agent. Gartner noted, as reported by W.Media, that many use cases labelled agentic do not require an agentic implementation. A pilot is allowed to show that a simple rule, a form or a better template is enough.
How do you get started?
Start with a shortlist of three to five candidate processes and rate them on frequency, data availability and pain. Pick the one whose data is already recorded and accessible, not the grandest vision, and write its charter before anyone chooses a tool.
A practical first two weeks:
- Build the shortlist with the business teams and start a baseline measurement for the two strongest candidates.
- For the chosen process, write down hypothesis, scope, metrics and thresholds.
- Bring in the decision-maker, the business owner and employee representatives; sort out data access.
- Run a short feasibility check on a handful of real, anonymised cases to rule out obvious technical blockers.
- Only then build the pilot and launch it with an end date.
In our own operations, one rule has proved its worth above all: the end date is in everyone's calendar before the first model call is made. For an overview of which agent types fit typical processes in mid-sized companies, see AI agents. If you want to build AI capability across the organisation more broadly, the page on a fractional head of AI explains that route. Terms such as human-in-the-loop or hallucination are explained in the AI glossary.
Frequently asked questions
How long should an AI pilot run?
Long enough for enough real cases to pass through to test the hypothesis, and no longer. For many office processes that means a few weeks; what matters is that the end date is fixed before launch.
What is the difference between a proof of concept and a pilot?
A proof of concept shows with sample data that something is technically possible. A pilot tests in live operations, with real users, whether it pays off and gets adopted.
Who should decide on the outcome of a pilot?
A named person who owns the budget for the process, not the team building the system. They sign off the success criteria before launch and decide against those criteria at the end.
Do employee representatives need to be involved?
If the system can record how individual employees work, consultation may be required depending on the jurisdiction, and in Germany works councils have co-determination rights. Involve them early; this is orientation, not legal advice.
Is a stopped pilot a failure?
Not if it stops on criteria agreed in advance. A cleanly ended pilot gives you a reliable no and avoids the cost of a rollout that would not have paid off.
Sources
- Intelligent CIO: Gartner predicts 30% of generative AI projects will be abandoned after proof of concept by end of 2025 (August 2024) intelligentcio.com
- W.Media: Over 40 percent of Agentic AI projects could face the axe by end of 2027: Gartner (June 2025) w.media
- Anthropic: Define your success criteria (Claude documentation) platform.claude.com
- NIST: AI Risk Management Framework nist.gov
- EU AI Act, Article 4: AI literacy artificialintelligenceact.eu
- German Works Constitution Act, Section 87: Co-determination rights gesetze-im-internet.de