Measuring customer service quality when AI answers first
By Keith Hamilton · · 4 min read
Measuring AI customer service properly matters because when an AI agent answers your calls or chats first, the usual service figures look better than the service really is. Calls are picked up, queues disappear and average handling time falls, yet some customers are still ringing back or giving up. This article explains which AI customer service metrics tell you whether customers are actually being helped, how to review AI conversations consistently, and how to avoid the figures that flatter.
Is an answered call the same as a resolved one?
No. Treating "answered" as "helped" is the first trap. An AI agent can answer every call, but that only tells you the phone was picked up.
We see this distinction in our own work. For an SME telecoms business running several companies, an AI voice agent took the share of inbound calls answered from 23% to virtually 100%. That is a real improvement, but it measures calls answered, not queries resolved. Resolution has to be measured separately, and so does what happens to the calls the agent passes on.
Which AI customer service metrics matter?
Six AI customer service metrics matter, and they work best read together rather than one at a time.
Answer rate
The share of calls or chats answered rather than missed or abandoned. This is where AI makes the biggest visible difference.
AI resolution rate
The share of conversations fully dealt with by the AI, without a transfer. Be clear about how you decide a conversation was resolved. "The customer did not ask for a person" is not the same as "the customer got what they needed".
Repeat contact
Customers who get in touch again about the same thing within a few days. This is one of the most useful checks on resolution, because it exposes answers that sounded complete but did not solve the problem.
Transfer rate and reasons
How many conversations go to a person, and why. The reasons show where the AI is missing information, misunderstanding customers or being asked things it should not handle.
Time to a person
When a customer is transferred, how long they then wait. A fast AI followed by a long queue is not a good experience.
Accuracy
The share of reviewed conversations where the AI gave correct, current information. This only comes from reading or listening to conversations.
How do you calculate a true AI resolution rate?
A true resolution rate removes the "resolved" conversations where the customer came back about the same thing. Here is a worked example.
An AI agent handles 1,000 calls in a month. The system reports that 500 ended without a transfer, so the resolution rate looks like 50%.
Of those 500 callers, 100 rang back within seven days about the same thing. Those calls were not resolved. The true figure is 400 out of 1,000, or 40%.
Both numbers are worth knowing, but only the second tells you how often customers were actually helped first time.
How should you review AI conversations?
Review AI conversations with a fixed routine, because reading transcripts at random without a method tells you little.
- Take a regular sample. Pick a set number of conversations at random each week, plus every one that was flagged, complained about or transferred for an unusual reason
- Use a short scoring sheet. For example: was the information correct, was the customer's question understood, was the tone right, was the hand-over timely, and was anything said that should not have been
- Have two people score a few of the same conversations now and then, to check you are judging them consistently
- Record what you change as a result, so you can see whether it helped
Our article on blending AI and people in a contact centre covers how this fits into the wider running of a team.
Which figures mislead?
Some measures look good for the wrong reasons:
- High resolution with rising repeat contacts: customers are giving up on the AI, not getting answers
- Low transfer rate: this sometimes means it is hard to reach a person, not that the AI is doing well
- Short conversations: these mean either quick answers or customers hanging up in frustration, and only listening tells you which
- Satisfaction surveys alone: the customers who answer them are rarely a fair sample
When a figure improves sharply, check a sample of the conversations behind it before you celebrate.
Why do you need a baseline?
You need a baseline because without a starting point there is nothing to compare against. Before the AI goes live, record answer rate, repeat contacts, complaints and, where you can, a reviewed sample of human-handled conversations scored on the same sheet.
That lets you judge customer service quality with AI against the same standard as before, and it is the basis of any honest AI business case.
Who should act on the results?
One named person should own the weekly review and act on it. Measures are only useful if someone acts on them.
Make sure what they find leads to changes in the AI's information, its rules or its hand-over to your team. Involve your customer-facing staff, who spot problems in the figures before anyone else.
Where Forwardcycle fits
Forwardcycle helps UK businesses set a baseline, choose honest measures and build a review routine before AI starts answering customers. Find out more about our customer experience services or take the free AI Readiness Assessment.