How to run a 30-day AI pilot that gives a clear answer
By Keith Hamilton · · 4 min read
An AI pilot is a small, time-limited test that answers one question: does this use of AI work well enough in our business to be worth keeping? Done well, it gives you a clear yes or no for a modest outlay; done badly, it drifts on for months and leaves everyone unsure. This article sets out how to run an AI pilot over 30 days in a small or medium-sized business, and the mistakes that stop a pilot from giving you an answer.
Which task should an AI pilot test?
An AI pilot tests one narrow task, done by a small number of people, that happens often enough to give real results within a month. Pilots fail when they try to do too much. Whether you call it a pilot or an AI proof of concept, a small business gets the clearest answer from a single task.
Testing AI in your business works best on tasks that are repetitive, frequent and low risk:
- Drafting first replies to routine enquiries
- Sorting a shared inbox
- Pulling details out of documents
- Answering a common customer question
If you have not chosen yet, our guide on where to start with AI shows how to build and rank a list. Avoid tasks that touch sensitive personal data, such as health or financial difficulty, for a first pilot. Leave those until you have the right safeguards in place.
How do you define success before the pilot starts?
Define success by writing one sentence the pilot must answer and agreeing the numbers that count as a pass. For example: can AI draft first replies to delivery enquiries that staff can approve with little or no editing?
Typical success measures are a reduction in time spent on the task, a faster response time, or a share of drafts approved without changes. Agree a level below which you will stop. Writing this down first stops the goalposts moving once results start to arrive.
Name one person who owns the pilot. It does not need to be a technical person, but it does need to be someone with time to keep it moving.
What should you measure before day one?
Measure how the task performs today, for a week or two before the pilot starts. You cannot show an improvement without knowing where you started. Record:
- Hours spent on it each week.
- Volume: how many emails, calls, documents or requests.
- Speed: how long customers or colleagues wait.
- Quality: how often things need correcting.
Simple tallies in a spreadsheet are enough. Our article on the real cost of manual work shows how to turn those hours into a cost.
What happens in week 1 of a 30-day AI pilot?
In week 1, set the tool up and run it in the background while people carry on doing the work as normal. The AI produces its output, but nothing reaches a customer. Compare what it produced with what your team did.
This shows you quickly where the tool is weak, which instructions need adjusting and which kinds of case it should not handle at all. It is far cheaper to find that out now.
What happens in weeks 2 and 3?
In weeks 2 and 3, the tool does the work and a person reviews its output before it is used. Ask the people involved to note every correction and anything that surprised them. Keep the notes simple: what was wrong, and how long it took to fix.
Hold a short check-in at the end of each week. Small adjustments are fine. Changing the task or the success measures is not, because then you lose the comparison with your baseline.
How do you decide at the end of week 4?
At the end of week 4, put the same measures as your baseline side by side and decide to keep, adjust or stop. Include the time people spent checking and correcting, because that is a real cost.
A team that spent 8 hours a week on a task before the pilot and 5 hours a week during it, including checking time, saves 3 hours a week. Over 52 weeks, that is 156 hours a year. Multiply those hours by the loaded hourly cost of the people involved to put a value on them, then set that value against the running costs of the tool.
Then make one of three decisions:
- Keep and extend if it met the success measures.
- Adjust and rerun if it came close and you know why it fell short.
- Stop if it did not work. That is a useful result, reached cheaply.
If you need to take the result to a board or partners, our guide to building an AI business case shows how to present it.
How do you report pilot results accurately?
Report each number for exactly what it measures. In our work with an SME telecoms business, inbound calls answered rose from 23% to virtually 100%. That measures whether a call was picked up, not whether the caller's query was resolved by the AI.
Both are worth knowing, but they are different things. A pilot report must say which one it is describing.
Why do AI pilots fail to give an answer?
AI pilots fail to give an answer for six avoidable reasons:
- The task was too broad, so results were mixed and hard to read.
- There was no baseline, so nobody could say whether things improved.
- Nobody owned it, so it stalled when people got busy.
- The success measures changed halfway through.
- The data the tool needed was incomplete or out of date.
- Staff were not told why it was happening, so they worked around it.
Where Forwardcycle fits
Our Quick-Win Sprint is a fixed-scope piece of work over 4 to 6 weeks that delivers one to three automations, each measured before and after. If you are not yet sure which task to test, the AI Opportunity Diagnostic sizes your options first. For a quick starting point, take the free AI Readiness Assessment or arrange a free call.