Measuring AI results: evidence your board will believe
By Keith Hamilton · · 4 min read
Once AI is in use, the board will ask a fair question: is it working? Measuring AI results properly is how you answer with evidence, because supplier dashboards and enthusiastic team feedback are not enough. It means choosing the right measures, being exact about what each one shows, and comparing like with like. This guide explains how.
Which AI performance metrics matter?
Measure outcomes, not activity. Many AI tools report activity: messages handled, documents processed, minutes of conversation. Those numbers are useful for running the tool, but they do not tell the board whether the business is better off.
Link each measure back to the reason you started the work. If the aim was to release staff time, measure hours. If it was to improve service, measure response times, complaints or customer outcomes. If it was to recover money, measure what was recovered.
Keep the set short. Three or four AI performance metrics cover most pieces of work:
- Volume: how much work went through
- Time: staff hours spent, or time taken to respond
- Quality: errors, rework, complaints or escalations
- Cost: running costs and internal time, set against the benefit
What does each AI result actually show?
Each figure shows one specific thing, so state exactly what it covers. A measure can be accurate and still misleading if nobody is clear about its scope.
Take our own published case study. For an SME telecoms business running several companies, inbound calls answered rose from 23% to virtually 100% after an AI voice agent was introduced. That is a real and important change. But it measures whether a call was picked up, not whether the caller's query was resolved by the AI. Anything the agent cannot resolve goes to the right team at head office.
Saying this plainly does not weaken the result. It makes it more credible, because the board knows exactly what it is looking at. If resolution matters too, it needs its own measure.
For every figure you report, write one line on what it does and does not show.
How do you set an AI baseline?
Record the baseline before anything changes, using a method you will repeat afterwards. A before-and-after comparison is only fair when both sides are measured the same way.
If you counted hours from timesheets before, count them from timesheets afterwards, not from estimates. If you measured response times over a full month before, measure a full month afterwards. Our guide to building an AI business case covers how to set a baseline.
How do you allow for other things that changed?
Compare against more than one period and record other changes alongside the results. AI is rarely the only thing that changes in a quarter: demand rises and falls with the seasons, staff join and leave, prices change. Any of these will distort a result if you ignore it.
Four simple checks:
- compare the same period last year as well as the period just before
- where you can, compare a team or site using the tool with one that is not yet
- record other significant changes alongside the results
- look at the trend over several months rather than one good week
You do not need a formal study. You need enough care that the board trusts the comparison.
How do you count the hidden time?
Subtract the time people spend checking AI outputs, correcting mistakes and handling exceptions. In the early weeks that time is real, and it belongs in the results.
A worked example
An administrator spends 10 hours a week sorting a shared inbox. With AI triage, sorting takes 3 hours a week and checking the AI's categorisation takes another 2 hours.
- The honest saving is 10 − 3 − 2 = 5 hours a week, not 7.
- When the checking time falls to 1 hour, the saving becomes 10 − 3 − 1 = 6 hours.
Report both figures, and explain why they changed.
Is time saved the same as cash saved?
No. Hours released are capacity, and they become a cash saving only if they reduce overtime, agency spend or the need to hire, or if the time goes on work that brings in money.
Blurring the two undermines any AI ROI measurement, and boards notice. Report released hours as hours, put a value on them using a stated hourly cost, and say clearly what the time is now being used for.
Should you report what went less well?
Yes. When reporting AI results to the board, include the measures that did not move, the problems you found and what you did about them, because a report with only good news will not be believed for long.
Honest reporting builds a track record. Once the board has seen it on small pieces of work, it is far more willing to approve larger ones.
Where Forwardcycle fits
Every Quick-Win Sprint we deliver is measured before and after, and our commercial leadership service reports progress to the board in pounds and hours. If you are still deciding where AI could help, start with the free AI Readiness Assessment.