Safe AI customer service automation comes down to five patterns you can defend to Legal

Safe AI customer service automation is mostly a question of what you ask for before you sign. The model will be wrong sometimes. Every vendor's will, including ours. What separates a quiet rollout from a story in the trade press is whether the AI was boxed in: told where its answers may come from, when to stop guessing, what it may touch, what gets logged, and when a person takes over. Five patterns. What follows is the version you can hand to Legal. If you only want the checklist or the vendor comparison, they're further down.

You're right to be cautious#

Vendors will show you a demo where the AI handles a returns question flawlessly. What they won't show you is the customer who asks something the AI wasn't trained on. Or phrases it oddly. Or pastes in an angry email threatening legal action.

When you bring AI to procurement, the same objections come up every time:

• What if it invents a policy that doesn't exist?
• What if it says something that damages the brand?
• Can we prove what it said if a regulator asks?
• What happens when it hits something it shouldn't be handling?

None of those is hypothetical. Each has already happened to a company with more engineers than you.

What happens without guardrails#

Cursor, April 2025#

A Cursor user kept getting logged out when they switched machines, so they emailed support. The reply, signed "Sam", said this was "expected behavior" under a new login policy. There was no such policy. "Sam" was an AI, and it had filled a gap in its knowledge with something that sounded like a rule (Fortune, 19 April 2025).

The Reddit post reached Hacker News under the title "Cursor IDE support hallucinates lockout policy, causes user cancellations" and collected 1,500 points and 600-odd comments. Co-founder Michael Truell replied on Reddit: "Hey! We have no such policy. You're of course free to use Cursor on multiple machines" (The Register, 18 April 2025). On Hacker News he wrote that "something very clearly went wrong here", that the user had been refunded, and that AI replies in email support were "now clearly labeled as such" (Hacker News, 16 April 2025).

What was missing: a boundary on where answers may come from. When the bot had no approved source for the question, nothing stopped it inventing one.

Klarna, 2024 to 2025#

In February 2024 Klarna announced that its AI assistant had handled 2.3 million conversations in its first month, two-thirds of its customer service chats, doing "the equivalent work of 700 full-time agents" (Klarna, 27 February 2024). Over the following year it paused hiring and cut staff to shift towards AI.

Fourteen months later the chief executive, Sebastian Siemiatkowski, told Bloomberg: "As cost unfortunately seems to have been a too predominant evaluation factor when organizing this, what you end up having is lower quality." Klarna started recruiting support staff again, through a remote pilot aimed at students and professionals. His other line from the same interview is the one to pin above your desk: "I just think it's so critical that you are clear to your customer that there will be always a human if you want" (CX Dive, 9 May 2025).

Klarna didn't switch the AI off. It added a guaranteed route to a person, which is the thing the 2024 announcement never mentioned.

Air Canada, 2024#

Jake Moffatt's grandmother died. Air Canada's website chatbot told them they could buy a full-price ticket and claim the bereavement fare afterwards, "within 90 days of the date your ticket was issued by completing our Ticket Refund Application form". The airline's own policy page said the opposite: no retroactive bereavement fares. They flew, applied, and were refused. The full ruling is in our Air Canada chatbot case write-up.

At the British Columbia Civil Resolution Tribunal, Air Canada argued it couldn't be held responsible for what the chatbot said. Tribunal Member Christopher Rivers: "In effect, Air Canada suggests the chatbot is a separate legal entity that is responsible for its own actions. This is a remarkable submission." He found that Air Canada "did not take reasonable care to ensure its chatbot was accurate", noted that the airline never explained why customers should have to double-check one part of its website against another, and ordered it to pay $812.02 (Moffatt v. Air Canada, 2024 BCCRT 149, 14 February 2024).

The sum was small, and the principle wasn't: the chatbot is your website, and you're liable for it. What Air Canada couldn't do was point to a single source of truth and show why the bot had departed from it.

Commonwealth Bank, August 2025#

Nothing like the above happened at Australia's Commonwealth Bank. The bank put a voice bot on its inbound enquiries line and announced 90 redundancies, 45 of them in direct banking. Then it reversed the decision, saying its assessment "did not adequately consider all relevant business considerations and this error meant the roles were not redundant". It apologised to the staff and offered them their jobs back (AAP via The Canberra Times, 21 August 2025).

As far as anyone has reported, the bot didn't say anything wrong. The business case did. Which is the guardrail nobody puts in the RFP: the numbers you promised your CFO need a review date too.

Each failure had the same shape. The AI operated outside any defined boundary, filled a gap with something plausible, and the company found out from its customers. What's changed since 2024 is that the legal position has hardened. An October 2026 brief from Columbia Law School's Center for Law and the Economy and Protect Borrowers, written by a former FTC chief technologist and a former director of consumer protection at the DC attorney general's office, puts it plainly: "it is a bedrock principle that companies are generally liable for the conduct of their agents, and that includes AI chatbots". Their one-line summary: "There is no AI exception to these principles" (The Legal Risks Posed by AI Chatbots, October 2026). And since 2 August 2026, Article 50 of the EU AI Act has required that anyone interacting with an AI system is told so, at the latest at the first interaction (Regulation (EU) 2024/1689).

Five patterns that cover the risk#

The same things keep going wrong, which means you can stop them. Five patterns cover the main risks of customer-facing AI.

It took me a while to see that the five split into two jobs. Patterns 1 to 3 prevent incidents. Patterns 4 and 5 catch the ones that slip through and route them to a person. A vendor that only offers the first three will give you fewer incidents and no visibility when one happens. A vendor that only offers the last two will tell you about every problem and keep having them. You need both halves.

What to ask for, pattern by pattern#

One question per pattern, one red flag, and the part vendors don't volunteer.

1. Knowledge boundaries#

This is what would have stopped the Cursor incident. The AI answers only from your approved documents, FAQs and policies. Nothing from general training data about how companies usually do things.

Ask: "For any response, can we see exactly which document it came from?"

Red flag: No visible source attribution, or vague talk of "grounding" without actual citations.

Worth knowing: your help docs contradict each other. Everyone's do. Page A says refunds take 3-5 days, page B says 5-7, and the AI will surface every inconsistency you've been ignoring for years. I'd budget two to four weeks to clean the knowledge base before launch, and I'd treat that as the project, with the AI as the thing you switch on at the end of it. It's the difference between a successful pilot and an embarrassing one.

2. Confidence thresholds#

Think of a dial. The AI scores how sure it is about every reply, and below a threshold it stops and hands over instead of guessing.

Ask: "Can we set the confidence threshold ourselves? What's the default, and what happens below it?"

Red flag: The AI never says "I'm not sure" or "let me get someone who can help". A bot that's always confident is a bot with the dial removed.

Worth knowing: the number matters less than the fact that you can move it. In Hay.chat every answer that states facts is scored on how well it's grounded in the documents it retrieved, and the defaults are: 0.8 and above is sent. 0.5 to 0.79 triggers a recheck, where the agent pulls more documents and tries again. Below 0.5 goes to a person if Human Escalation is on; with it off, the reply is sent anyway, so leave it on. Both thresholds are a setting. Set it high and everything escalates, which means you've built an expensive routing system. Set it low and the guesses get through. Expect to adjust weekly for the first month, then rarely.

3. Action permissions#

Clear rules for what the AI may do on its own, what needs a person, and what it may never touch.

Ask: "Can we set different permissions for different action types? What's the most granular control you offer?"

Red flag: One on/off switch for all actions, or no record of which actions the AI actually took.

Worth knowing: there's a wide gap between an AI that answers questions and one that takes actions. An AI that can issue a €500 refund is a different product from one that looks up an order status, even if they share a chat window. Vendors blur this because action-taking AI is harder to build safely. If a vendor says their AI "does everything", ask three specific questions. Can it change a customer's account? Move money? Cancel an order? Vague answers mean they haven't worked it out.

One more distinction worth making in the room: an approval queue for replies is a different thing from an approval queue for actions. Hay.chat's approval mode, for example, holds AI messages for a person to approve, edit or block. Actions are limited separately, by which tools a playbook references and by what the integration allows (a Shopify refund can't exceed what Shopify says is refundable). Ask which of the two you're being shown.

4. Audit trails#

Every conversation logged with timestamps, the sources cited and the confidence score, and exportable in a format your Legal team can open.

Ask: "Can we export complete interaction histories? How long are logs kept? What exactly is logged?"

Red flag: Logs that hold summaries instead of messages, or retention shorter than your compliance requirements.

Worth knowing: your team will use audit trails daily, and regulators almost never. Debugging a strange answer. Spotting gaps in the knowledge base. Settling a dispute about what was said. I've watched teams lose whole afternoons to the last one when the logs only held summaries. Customers are starting to expect the same visibility. In Zendesk's CX Trends 2026 survey, 95% of consumers said they expect an explanation for decisions AI makes about them, while only 37% said they currently offer one (Zendesk CX Trends 2026). A good audit trail is a productivity tool first and a compliance record second.

5. Escalation triggers#

This is the half of the problem Klarna fixed by hiring. Rules that route a conversation to a person based on sentiment, topic, customer value, specific phrases, or the AI's own confidence.

Ask: "When a conversation escalates, does the agent see the full history? How fast is the handoff?"

Red flag: Escalation adds a queue and a delay, or the agent arrives blind.

Worth knowing: escalation is the system working. If you're pushing for a single-digit escalation rate, you're probably keeping the wrong conversations automated, and the customers in them know it. I don't have a benchmark figure I'd defend, because it depends on what your tickets are. What I would defend is this: a handoff without context is worse than no AI at all. If the customer has to repeat everything to the human, you've made their day worse and paid for the privilege. One commenter on the Hacker News thread about Klarna summed up the alternatives: "Humans, or an endless redirect loop." Context transfer matters more than handoff speed. There's more on writing the rules in our escalation management entry.

The sixth thing: tell people it's an AI#

For EU customers this stopped being a style choice on 2 August 2026. Article 50 of the EU AI Act says providers of AI systems "intended to interact directly with natural persons" must make sure those people "are informed that they are interacting with an AI system, unless this is obvious", and the information has to arrive "at the latest at the time of the first interaction" (Regulation (EU) 2024/1689, Article 50). Cursor added labels to its AI email replies the week of its incident, which is the right move and also the floor. As one Hacker News commenter asked at the time, what is a customer supposed to do with a label, other than trust the reply less? Disclosure plus a visible route to a person is the version that works. Check that your vendor lets you set both.

Your evaluation checklist#

Bring this to the next vendor conversation. Print it. Scrolling a PDF while someone's demoing at you is awkward.

Priority: ●●● Essential | ●●○ Important | ●○○ Nice to have

The first 30 days are mostly setup. Most of your time goes on cleaning the knowledge base, tuning the threshold and writing escalation rules, and the AI gets better as real conversations come through and you adjust. If you expect magic on day one you'll be disappointed by day three. Treat the first month as training.

How Hay.chat compares#

We're the new kids. Fin (formerly Intercom Fin) and Zendesk have been at this longer, with bigger teams and more integrations. This is where we differ on the five patterns, checked against each vendor's public pages on 20 October 2026. Competitor prices are as listed in US dollars.

Two things aren't in the table because we couldn't verify them from either competitor's public pages: whether a person can approve every AI reply before it's sent, and what a conversation export contains. Ask them. In Hay.chat, approval mode holds each AI reply for a team member to approve, edit or block, and a conversation exports to PDF or CSV with every message, timestamp and the metadata behind it.

Why source-available matters#

Your security team can read how customer data is processed instead of taking a questionnaire's word for it. If we disappear tomorrow, you can run the code yourself. For regulated industries, or companies with strict vendor policies, that's often what moves AI from "not approved" to "we can work with this".

Where we fall short#

We haven't started SOC 2 or ISO 27001. AI replies are generated by OpenAI in the US under EU Standard Contractual Clauses, and EU inference is planned rather than live. Both facts, and the rest, are on our security page. For the full pricing comparisons see Hay.chat vs Fin and Hay.chat vs Zendesk AI. If you'd rather check the thresholds, approval mode and the export for yourself, start a 30-day free trial. No credit card.

What to actually expect#

No vendor can guarantee zero hallucinations. Not us, not anyone who's being straight with you. Gartner predicts that agentic AI will resolve 80% of common customer service issues without a human by 2029 (Gartner, March 2025), and, three months later, that more than 40% of agentic AI projects will be cancelled by the end of 2027 for "escalating costs, unclear business value or inadequate risk controls" (Gartner, June 2025). Both can be true. The tooling is getting there. Most buyers' controls aren't. Forrester's Kate Leggett expects service quality to dip in 2026 as companies wrestle with the complexity of deployment (Forrester, November 2025). Realistic looks like this:

Mistakes will happen. What separates the companies in this guide from the ones you never read about is how fast they caught the mistake and fixed it. That's what the five patterns are for.

Sources#

• Cursor: Fortune, 19 April 2025; The Register, 18 April 2025; Michael Truell on Hacker News, 16 April 2025
• Klarna: press release, 27 February 2024; CX Dive, 9 May 2025, reporting Siemiatkowski's Bloomberg interview of 8 May 2025
• Air Canada: Moffatt v. Air Canada, 2024 BCCRT 149, 14 February 2024
• Commonwealth Bank: AAP via The Canberra Times, 21 August 2025
• Law: Regulation (EU) 2024/1689 (the AI Act), Articles 50 and 113; Nguyen and Teitelbaum, The Legal Risks Posed by AI Chatbots, Columbia Law School Center for Law and the Economy and Protect Borrowers, October 2026
• Analysts: Gartner, 5 March 2025 (via CMSWire); Gartner, 25 June 2025 (via Outlook Business); Forrester, Predictions 2026, 10 November 2025; Zendesk CX Trends 2026
• Vendor pages, all checked 20 October 2026: Fin pricing; Fin regional data hosting; Fin answer sources; Fin Tasks and Procedures; Zendesk pricing; Zendesk generative reply settings; Zendesk actions in AI agents; Zendesk Data Center Location add-on; Zendesk Regional Data Hosting Policy
• Hay.chat: guardrails, settings, conversations, documents, data retention and the security page