A customer service quality assurance rubric that scores outcomes, not scripts
Six categories weighted 60/40 to outcomes, a calibration routine so the scores mean something, and a four-week rollout
Published · Updated
Customer service quality assurance only works when the rubric asks one question before any other: did we solve the problem and keep the customer? Most scorecards never ask it. They ask whether the agent used the customer's name, followed the greeting script and signed off properly, and then everyone wonders why the agents with the best QA scores have the worst retention.
I've run support teams for over a decade and I've watched QA programmes die the same way more than once. Someone builds a scorecard with too many categories (Zendesk's own guide puts the average at 14, with a median of 8). Reviewers spend half an hour per ticket. Scores go into a spreadsheet. Nothing changes. Three months later the spreadsheet stops getting updated and QA becomes a word people say in interviews.
This playbook is the version that stuck. It gives you a six-category rubric weighted 60/40 towards outcomes, the calibration routine that stops your 4 being someone else's 3, the reviewer biases that will quietly wreck your numbers, and a four-week rollout. No email gate, no download. The whole thing is on this page, and if you only want the scorecard, skip to the rubric.
Why most customer service QA programmes fail#
They measure what's easy, not what matters#
Process metrics are seductive because they're objective. "Did the agent use the customer's name?" has a clear answer. "Did the customer leave feeling the problem was handled?" needs judgement. So rubrics drift towards whatever can be ticked: greetings, sign-offs, script compliance, hold time.
You can see the drift in the data. The most popular rating categories, according to the benchmark report Zendesk cites in its QA programme guide, are solution, grammar, tone, empathy, personalisation, following internal processes and going the extra mile. One of those seven is the outcome. Five are process.
Customers don't care about the greeting. In 2010, Matthew Dixon, Karen Freeman and Nicholas Toman published a study of more than 75,000 customers in Harvard Business Review and found that 20% of the "satisfied" ones intended to leave anyway, and that customers were "four times more likely to leave a service interaction disloyal than loyal". Effort predicted loyalty better than satisfaction did. That paper is where the customer effort score comes from, and nothing I've seen since has overturned the basic finding. An agent who says "Hey, what's up?" and fixes the problem in two minutes beats one who delivers the perfect scripted greeting and doesn't.
Nobody calibrates the reviewers#
Without calibration, your 4 is someone else's 3. I've seen teams where the same conversation scored a full point apart depending on who reviewed it, which means the scorecard was measuring reviewer mood as much as agent performance. Agents work this out fast, and once they do, they stop trusting the scores.
The fix is simple and people skip it anyway: monthly sessions where reviewers score the same conversations independently, then compare. My working target is 80% of scores landing within one point of each other. That's a rule of thumb, not a research finding. Vendor guides quote anything from 80% to 85% and none of them show their data. When you're starting, 70% is fine. What matters is that the number goes up.
The scores never turn into coaching#
If you're not coaching off the scores, stop scoring. You're wasting everyone's time. The benchmark report Zendesk cites says 44% of teams use conversation-review feedback in one-to-ones and coaching sessions. Read that the other way round: more than half of the teams doing QA aren't doing anything with it.
Enough about failure.
The outcomes rubric#
Six categories, weighted towards outcomes, built to be scored in 10 to 15 minutes by someone who knows your product and your customers. It skips a lot on purpose and keeps what moves retention.
A note on the 60/40 split, because people ask: there's no study behind it. It's my recommendation, from running support teams and watching which scorecards agents respected and which ones they gamed. If your rubric weights process and outcomes equally, you're over-weighting process. Regulated industries may need a heavier process weight. Start at 60/40 and move it when your own data tells you to, not before.
| Category | Weight | The question it asks |
|---|---|---|
| 1. Resolution quality | 25% | Did we fix it? |
| 2. Customer effort | 20% | How hard did the customer have to work? |
| 3. Retention signal | 15% | Are they staying? |
| 4. Accuracy and knowledge | 20% | Was what we told them true? |
| 5. Communication and empathy | 15% | Did they feel helped, or processed? |
| 6. Process adherence | 5% | Did we follow the rules that exist for a reason? |
Outcomes (60% of the score)#
Three categories that ask: what happened for the customer?
1. Resolution quality (25%)#
The most important category. A customer contacts support because something is wrong. Did we make it right? Everything else is secondary.
| Score | Criteria |
|---|---|
| 5 | Issue fully resolved in this interaction. Customer confirmed or clearly indicated satisfaction. |
| 4 | Issue resolved but minor loose ends remain (for example, "tracking email coming in 24 hours"). |
| 3 | Core issue addressed but secondary concerns left open. Acceptable for complex cases. |
| 2 | Attempted resolution but the issue persists. Customer will need to contact again. |
| 1 | No meaningful progress. Issue unaddressed, or the agent's actions made things worse. |
Some issues can't be resolved in one contact. Shipping delays, third-party dependencies and complex bugs take time. Score whether the agent did everything within their control and set clear expectations. A 3 is right when resolution depends on factors outside the agent's influence. If you track first contact resolution as a team metric, don't double-count it here.
2. Customer effort (20%)#
The customer effort score exists because that 2010 study found effort predicted loyalty better than satisfaction did. When customers have to repeat themselves, get transferred, or do the agent's research for them, they remember. And they leave.
| Score | Criteria |
|---|---|
| 5 | Effortless. Agent anticipated needs and took initiative. Customer gave their information once. |
| 4 | Low effort. Minimal clarification needed. Conversation flowed. |
| 3 | Moderate effort. Some back-and-forth, but justified by the complexity of the issue. |
| 2 | High effort. Customer repeated information, or faced unnecessary transfers or holds. |
| 1 | Customer had to diagnose, research or solve parts of the problem themselves. |
Security verification takes effort by design. Don't penalise agents for identity checks, two-factor prompts or compliance steps. Score the effort beyond what security required.
3. Retention signal (15%)#
This one's fuzzy, on purpose. You're reading signals, and signals are all you'll get. Some customers never say thank you, others complain loudly and stay for years. Focus on explicit statements and clear shifts in sentiment.
| Score | Criteria |
|---|---|
| 5 | Explicit positive signal: gratitude, stated intent to continue, or sentiment clearly improved. |
| 4 | Positive close. No churn indicators. Customer's final message suggests satisfaction. |
| 3 | Neutral. Issue handled but the customer's sentiment is unchanged from the start. |
| 2 | Warning signs: customer mentioned competitors or expressed unresolved frustration. |
| 1 | Explicit churn signal: threatened to leave, asked to cancel, or escalated the complaint. |
A 1 here doesn't mean the agent failed. Some customers decided to leave before they got in touch, and a graceful goodbye is still a 1 on the signal. Score the signal, then note separately whether recovery was ever realistic.
Process (40% of the score)#
Three categories that ask: how did we get there? Process matters because it's what you can train.
4. Accuracy and knowledge (20%)#
Wrong information erodes trust faster than almost anything else. A customer who gets the wrong return window or the wrong price will remember, and they'll tell people.
| Score | Criteria |
|---|---|
| 5 | All information accurate. Agent showed strong product knowledge and gave proactive guidance. |
| 4 | Accurate on the key points. Minor details missing but nothing that affected the resolution. |
| 3 | Mostly accurate. One minor error that didn't meaningfully affect the customer. |
| 2 | Error given but caught and corrected during the conversation. |
| 1 | Significant error given and not corrected. The misinformation affected the outcome. |
Policy often has legitimate grey areas. If an agent reads it in the customer's favour and the reading is defensible, that's not inaccuracy. Save the low scores for information that was objectively wrong.
5. Communication and empathy (15%)#
Customers can tell when they're being processed rather than helped. Empathy here means acknowledging the situation before jumping to the fix. Nothing more elaborate than that.
| Score | Criteria |
|---|---|
| 5 | Exceptional. Natural language, genuine empathy, tone matched to the customer's state. |
| 4 | Good. Professional, warm, acknowledged the customer's situation directly. |
| 3 | Adequate. Professional but noticeably templated. Got the job done. |
| 2 | Needs work. Tone mismatch, dismissed the customer's concerns, or felt robotic. |
| 1 | Poor. Dismissive, condescending, or made the customer feel worse for getting in touch. |
On tone: match the customer's energy and leave their negativity alone. An agent who stays composed while a customer shouts has earned a high score.
6. Process adherence (5%)#
Weighted low on purpose. Process should serve outcomes. But some processes exist for good reasons, compliance and security especially.
| Score | Criteria |
|---|---|
| 5 | Full compliance. Escalation paths followed, documentation complete, SLAs met. |
| 3 | Minor gaps. Key procedures followed but documentation incomplete or small steps missed. |
| 1 | Major violation. Required compliance steps skipped, SLA breached, or policy broken. |
If you're in finance, healthcare or legal, consider raising this to 10-15%. Compliance failures carry disproportionate risk in regulated industries.
That's the rubric. Using it is harder than building it: you have to score the same way every week, and then let the scores change how people work.
What nobody tells you about QA scoring#
Rubrics don't score conversations. People do, and people bring moods and assumptions to the job that they don't notice. Reviewers learn the next four the hard way.
Your brain will sabotage your scores#
You'll anchor on your first score#
Start with a 5 on resolution and you'll go easier on communication. Start with a 2 and you'll be harsher all the way down. Tversky and Kahneman described anchoring in 1974, and QA reviewers are not exempt.
Fix: randomise the order you score the categories, or do a quick high/medium/low pass across all six before you commit to numbers.
Your mood will leak into the scores#
End of the day, after a bad meeting, hungry: scores drop. My own afternoon scores run lower than my morning ones, and it took me a while to notice.
Fix: batch QA when you're in a neutral state. Never score straight after a difficult conversation of your own.
You'll score the conversation you would have had#
"I'd have offered a refund there" turns into a penalty for an agent who did something different that worked.
Fix: ask whether the agent's approach worked. Different isn't wrong.
One bad moment will colour everything#
An awkward line in message 3 and you'll read messages 4 to 10 looking for more.
Fix: score each category only on what's relevant to it. A tone slip doesn't make the product knowledge wrong.
The "3" problem#
This one will undermine the whole programme if you ignore it.
On a 1 to 5 scale, 3 should mean "met expectations". Did the job. Fine.
But most teams accidentally build a culture where 3 feels like failure. Agents get anxious about 3s. Managers treat 3s as problems to raise. Years of exam marking and performance reviews have taught everyone that anything below a 4 is a quiet warning, and that only low scores get discussed.
So reviewers inflate to avoid giving "bad" 3s, agents stop trusting the scores, and the rubric means nothing. The 3 problem has killed more QA programmes than anything else I've seen.
Fix: say it out loud, to the team. "A 3 means you did your job. We're looking for patterns of 2s and celebrating 5s. A run of 3s means you're solid." Say it again the next week. The first time you give someone a 3 to their face you'll feel bad about it. Say the words anyway.
Putting it into practice#
Sampling: make your 2% count#
Zendesk says the average team manually evaluates about 2% of support interactions, citing its Customer Service Quality Benchmark Report. Most teams I talk to are nearer 1%. Jim Iyoob, who has spent 33 years in call centres and BPOs, put it bluntly in CMSWire in March 2026: "Most QA programs in contact centers evaluate 2 to 5% of interactions. I will say that plainly: that is not a QA program. That is a sample."
He's right, and you still have to start somewhere. The goal is a sample that's representative enough to catch patterns.
- Baseline: 4 to 5 conversations per agent per week. That's my number. I haven't found an industry standard with data behind it. It gives you roughly 200 data points per agent a year, enough to see a trend.
- Prioritise the edge cases: long conversations, escalations, refund requests, low CSAT. That's where process breaks.
- Mix channels if you run chat, email and phone. Performance varies by channel.
- Keep 2 to 3 purely random tickets a week to catch what the targeting misses.
Reality check: 5 reviews across 20 agents is 100 reviews a week. At 10 to 15 minutes each, that's 17 to 25 hours. If that made you wince, you're not alone, and it's why 32% of teams now use AI to pick which conversations to review (same benchmark report). Without dedicated QA staff you either cut the sample or let the tooling do the triage. Hay.chat does a version of this for its own AI replies: each one gets a confidence score, anything below 0.5 goes to a person if you've switched human escalation on (otherwise the customer gets a fallback message), and the dashboard lists the conversations waiting for a human. That's a review queue you didn't have to build, although it won't score your rubric for you.
Calibration: make scores mean something#
- Weekly, 15 minutes: a quick stand-up on edge cases. "How would you score this one?"
- Monthly, one hour: everyone scores the same three conversations independently, then compares. Aim for 80% within one point.
- Quarterly: revisit the rubric itself. Has the product changed? Have the complaint patterns?
The monthly session will get tense sometimes. That's fine. The disagreements are where you find out what the rubric actually means.
Closing the loop: scores to action#
- Coaching trigger: a 2 or below in any category means a coaching conversation that week. One category at a time.
- Pattern recognition: one low score is noise. Three weeks of decline is a signal.
- System issues: low accuracy across several agents usually means a gap in the knowledge base. Fix the system before you coach five people for the same thing.
- Recognition: share the 5s. Examples teach better than rules.
Our customer service KPIs post covers which of these numbers predict churn and which just fill a dashboard.
Scoring the AI's replies too#
If an AI agent answers some of your conversations, its replies go through the same rubric. Gartner's February 2026 survey of 321 service leaders found 91% were under pressure from executives to implement AI, and the fastest way to lose that argument internally is to have no idea how good the AI's answers are.
Two things change. The sample problem gets worse, because the AI handles volume. And the weight shifts: communication and empathy gets less interesting, accuracy and knowledge gets more. Ty Givens, founder of CX Collective, told CX Today in August 2026 that AI scoring tools still struggle with nuance, so a person has to review the context and decide what matters. That's my experience too. Automated scoring widens coverage. It doesn't replace calibration, because now the thing being calibrated is a model.
The practical version while you're calibrating: hold the AI's replies for approval. Hay.chat's approval mode holds every AI reply for a team member to approve, edit or block before it's sent, and reviewers can rate each reply thumbs up or down with a comment, which gives you a running quality log. It holds replies only. A refund the playbook allows still goes through. Score a week of held replies against categories 1, 2 and 4 before you let anything auto-send. If you want to try that, the 30-day trial is free and doesn't ask for a card.
Your first four weeks#
You don't need all of this at once. The rollout below is the one that's worked for me.
Week 1: foundation#
Customise the rubric. Adjust the weights if you must, and add real examples from your own queue to each score level. Score ten conversations yourself to get a feel for it. Don't share those scores, they're practice. Pick two or three people who'll review regularly.
By Friday: a rubric you believe in and a named review team.
Week 2: calibration#
Hold the first calibration session. Pick three conversations that span the range, score them independently, then sit in a room and compare. You'll be tempted to skip this because it feels like overhead. Don't. Update the rubric from the disagreements. That's where the learning is.
Week 3: soft launch#
Review two or three conversations per agent. Don't share scores with agents yet. You're checking whether the rubric works: do the scores feel right, and what's missing? You'll find rough edges. That's what the week is for. Address the 3 problem with your reviewers now, before agents see a number.
Week 4: full launch#
Tell the agents. Explain the rubric. Say that 3 means good, more than once. Start sharing scores with written feedback, and book a coaching conversation for any 2.
After week 4#
Now it's consistency. Weekly quick calibrations, monthly deep ones, quarterly rubric reviews. Track whether scores line up with retention and CSAT, and adjust the weights when they don't.
The first four weeks are the hard part. After that it's just a habit.
Frequently asked questions
Sources#
- Matthew Dixon, Karen Freeman and Nicholas Toman, "Stop Trying to Delight Your Customers", Harvard Business Review, July-August 2010
- Amos Tversky and Daniel Kahneman, "Judgment under Uncertainty: Heuristics and Biases", Science 185(4157), 1974
- Zendesk, "How to measure and improve your Internal Quality Score", zendesk.com, September 2024 (the 2% and 44% figures, attributed to its Customer Service Quality Benchmark Report)
- Zendesk, "How to create a customer service QA program", zendesk.com, updated March 2026 (rating categories, the 14/8 scorecard figures and the 32% figure)
- Jim Iyoob, "Where Agentic CX Actually Pays Off (And Where It Doesn't Yet)", CMSWire, 23 March 2026
- Rhys Fisher, "Why AI-Only QA Still Leaves Support Leaders Doing the Heavy Lifting", interview with Ty Givens, CX Today, 28 August 2026
- Gartner, "Gartner Survey Finds 91% of Customer Service Leaders Under Pressure to Implement AI in 2026", press release, 18 February 2026
- Hay.chat product behaviour: confidence guardrails, message approval and feedback insights in the user guide
Score the AI's replies before customers see them
Hay.chat holds AI replies for your approval while you calibrate, and sends low-confidence answers to a person. Free for 30 days, no credit card needed.