Record a baseline before the pilot changes the work.
Sample enough recent cases to capture normal variation. Record elapsed cycle time, active staff time, handoffs, rework, errors and waiting. Note volume and case complexity so a quiet week is not compared with an unusually busy one.
Define the unit being measured: one enquiry, document, request or completed case. Without a stable denominator, totals can make improvement appear larger or smaller than it is.
Use a balanced set of measures.
Combine efficiency, quality and operating-control measures. A faster step is not useful if more cases need correction or staff stop trusting the result.
- Active handling time and end-to-end cycle time.
- Successful cases, corrections and unresolved failures.
- Exception rate and time spent on human review.
- Adoption: eligible cases actually processed through the pilot.
- Running cost per case at the observed volume.
Separate observed results from assumptions.
Keep a dated test log linking inputs, outputs, reviewer decisions and failures. Mark synthetic cases separately from live pilot cases. If the sample is small, report the count and range rather than presenting a precise percentage as a durable forecast.
Do not convert minutes from a short test directly into annual savings without accounting for supervision, maintenance, subscription costs and changing volume. State what was measured and what remains an estimate.
Decide whether to improve, expand or stop.
Review the pilot with users, the process owner and the person responsible for technical operation. Examine the awkward cases, not only successful examples. Decide which changes are required before more volume or broader access.
A pilot can be valuable even when it does not proceed. It may show that inputs need standardising, an integration is unreliable or human judgement is the essential service. Document the decision, ownership and next review date.
Questions before you start
Questions before you start
How long should an automation pilot run?
Long enough to observe representative volume and exceptions. The appropriate period depends on process frequency and seasonality, so agree a case target as well as a date range.
What counts as a successful AI output?
Define task-specific acceptance criteria and the required reviewer. Fluency alone is not evidence of correctness, completeness or fitness for the business decision.
Can a pilot prove return on investment?
It can provide inputs for an ROI decision, including observed effort, quality, cost and adoption. A limited pilot cannot guarantee future volume, behaviour or savings.