Topic AI and LLMs

Customer Service Chatbots: What to Automate, What Not

The short answer

A customer service chatbot works best on repetitive questions with one right answer: FAQs, order status, bookings and collecting details before a handover. Keep complaints, refunds, policy exceptions and anything with legal or financial weight with people. Ground answers in your own content, make handover to a person quick, and log what the bot can't answer.

Written by

Siddharth Kajaria, Founder and CEO

Published

Updated

Reading time

10 min read

At a hotel front desk a guest takes a key from a self-service rack while a staff member talks with another guest.
In this post10 min read
  1. Should you put a chatbot on customer service?
  2. What should a support chatbot automate, and what should stay human?
  3. Which kind of chatbot do you need?
  4. How do you stop a chatbot from making things up?
  5. How should the handover to a person work?
  6. How do you roll out a support chatbot?
  7. How do you measure whether a support chatbot is working?
  8. What does a support chatbot cost to run?
  9. Where else can AI help customer service?
  10. How we’d approach a support chatbot

Key takeaways

A chatbot on your support desk can answer the same twenty questions at any hour, or it can frustrate customers who only wanted a person. Which one you get depends less on the technology than on four decisions: what it is allowed to answer, where its answers come from, how it hands over to a person, and how you measure it. This guide covers each of them, and what drives the running cost.

Should you put a chatbot on customer service?

Put a chatbot on customer service when a large share of your incoming questions repeat, have one right answer and are already answered somewhere in your own content. Pull a month of tickets, emails or chats and sort them. If the same questions keep coming back (where is my order, can I change my booking, what is your returns policy), a chatbot can take them and give your team time for the conversations that need a person.

If most of your contacts are complaints, one-off problems or questions that need someone to look into an account and decide, a chatbot will mostly pass them on, and you pay for a step that adds little. Fix the help content or the self-service pages first.

What should a support chatbot automate, and what should stay human?

Automate the questions that are repetitive and have one right answer, and keep with people anything that needs judgement or carries risk. The line is not about how hard a question sounds. It is about what happens if the answer is wrong.

Automate

Repetitive, with one right answer

  • FAQs with stable answers, such as opening hours, delivery areas and returns policy
  • Order, booking or ticket status lookups
  • Appointment booking and rescheduling
  • Collecting a customer’s details and problem before a handover
  • Out-of-hours triage, so urgent issues reach a person first thing
Keep human

Needs judgement or carries risk

  • Complaints and upset customers
  • Refunds and exceptions to policy
  • Questions with legal, medical or financial consequences
  • Anything the bot can’t answer from your own content

Write this split down before any build starts, and make it the bot’s instructions. When a question falls on the human side, the bot’s job is to collect the details and pass it on, not to try.

Which kind of chatbot do you need?

You need a hybrid for most support work: fixed flows for tasks such as status lookups and bookings, AI for questions phrased in many different ways, and a person for everything else. A purely rule-based bot is enough when a handful of scripted tasks covers almost everything customers ask.

Rule-based, AI-powered and hybrid support chatbots compared
Compared onRule-basedFollows scripted flowsAI-poweredAnswers from your contentHybridRules for tasks, AI for questions
How it answersButtons, menus and keyword matches lead to answers someone wrote in advanceA language model reads the question and answers from the documents and pages you give itFixed flows for routine tasks; AI for open questions; a person for the rest
Set-up effortEvery flow and answer has to be written and kept up by handYour content has to be clean and current, and the answers testedBoth: the flows, the content and the rules for which one takes a question
Handles messy questionsPoorly: a question phrased in an unexpected way falls throughWell: the same question asked many ways gets the same answerWell, with the predictable tasks kept on fixed rails
Actions in your systemsReliable for the steps it was built for, such as a status lookupOnly through tools you connect and limitStatus lookups and bookings stay in fixed flows
How it failsLoops the customer back to the menuStates something your content doesn’t sayPicks the wrong path between a flow and an AI answer
Running costLow and flat per conversationModel usage per message, growing with conversation lengthModel usage only on the AI part
Best whenYou have a handful of tasks with fixed steps, and questions rarely stray from themQuestions are varied and the answers already exist in your help contentYou need both reliable lookups and answers to open questions

If you are weighing a chatbot against an agent that takes actions on its own, our guide to what an AI agent is sets out the difference.

Planning a support chatbot?

How do you stop a chatbot from making things up?

You stop a chatbot from making things up by making it answer only from your own content, show the source with every answer, and say plainly when your content doesn’t cover a question. The common technique is retrieval-augmented generation: the system finds the passages in your help articles and policies that match the question and asks the model to answer from those alone. Our guide to what RAG is explains how it works and why answers can still go wrong.

Four habits matter as much as the technique:

  • Refuse what’s out of scope. A support bot should not offer opinions, legal or medical advice, or answers about competitors’ products. Tell it what it may answer, and test that it declines the rest.
  • Keep the content current. A bot quoting last year’s returns policy is worse than no bot. Give one person the job of updating the content when policies, prices or products change.
  • Keep personal data out of it. Send the model only what a question needs, and keep details such as card numbers, ID numbers and health information out of prompts and logs where you can. Check what the model provider can see, how long your logs and theirs are kept, and the provider’s data-use terms for the plan you use, including whether your data may be used to train their models.
  • Log what it can’t answer. Every question it couldn’t answer is a gap in your content, and the list shows the team what to write next.

The NFTR Assistant we built (on the NITI Aayog Frontier Tech Repository portal) works this way: it answers from the knowledge base, cites its sources and reports unanswered questions to admins, so the gaps show up for the team to fill.

How should the handover to a person work?

The handover should be quick, available at any point, and pass the whole conversation to the person, so the customer never repeats themselves. It is the part of a support chatbot to design most carefully, because it is where customers decide whether the bot helped them or got in their way.

Decide three things in advance:

  • What triggers it. The customer asks for a person; the question is on the keep-human list; the bot has failed to answer twice; the customer sounds upset.
  • What the person sees. The conversation so far, the customer’s details and order or account, and the bot’s short summary of the problem.
  • What happens outside staffed hours. The bot says when someone will reply, takes the details and creates a ticket. It never pretends a person is about to join.

How do you roll out a support chatbot?

Roll it out in small steps: pick the repeating questions, fix the content behind them, design the handover, test on real past questions, then launch on one channel and review every week.

  1. Pick the questions

  2. Ground it in your content

  3. Design the handover

  4. Test before launch

  5. Launch small, log the gaps

  6. Review every week

Testing deserves its own plan before launch. Keep a set of real questions with the right answer and source for each, add the questions it should refuse, and run every version of the bot against the whole set, so you see when a change helps one answer and quietly breaks another. If you plan to build it yourself, start with how to build a chatbot in Python.

How do you measure whether a support chatbot is working?

Measure a support chatbot by how many questions it resolves without the customer coming back, and by how satisfied those customers are compared with customers your team helped. The deflection rate (the share of conversations it closed on its own) is the easy figure to report, and on its own it misleads: a customer who gives up on the bot also counts as closed.

Track these together:

  • Resolved without a person, and without a repeat contact. Count a conversation as resolved only if the same customer doesn’t come back about the same issue within a few days, by chat, email or phone.
  • Satisfaction after bot conversations. Ask one short question at the end, and compare the answers with the same question after conversations your team handled.
  • Handover rate and reasons. How often it passes to a person, and why. A rising “couldn’t answer” share means the content needs work.
  • Wrong answers. Read a sample each week and count any answer that states something your content doesn’t say. One is worth fixing that day.
  • The unanswered list. The questions it couldn’t answer, ranked by how often they come up. This is the week’s content plan.

What does a support chatbot cost to run?

A support chatbot’s running cost depends on how many conversations it handles, how long they run, how much of your content is sent with each answer, which model answers, and whether you built it or pay for a tool. People’s time to review conversations and update content belongs in the estimate too.

What drives the running cost of a support chatbot, and what to ask for instead
What drives itWhy it costs moreWhat to ask for
Number of conversationsAn AI chatbot pays for model usage on every message, so the bill rises with traffic, including spam and test chats.A cost per conversation measured in a pilot, and a daily cap with an alert.
Conversation lengthEach new message usually sends the conversation so far back to the model, so long chats cost more per message.Hand over to a person after a set number of turns, and summarise long histories.
Text sent with each answerPasting whole documents into every answer is an easy way for the bill to grow quietly.Send only the passages that match the question, with their sources.
The model chosenThe largest models cost much more per message, and greetings, status lookups and simple FAQs don’t need them.A smaller model for routine turns, a larger one only where tests show it is needed.
Build or buyBought tools are priced in different ways, so compare them at your own volume; a built one adds hosting and the work of keeping it current.Both priced at your expected volume for a year, including the people who review conversations.
Content upkeepThe bot is only as good as the content behind it. Someone has to update it when policies, prices or products change.A named owner for the content, and time in the plan each week to close the gaps the bot logs.

Where else can AI help customer service?

Beyond the chatbot, AI can help the people who answer customers. It can sort and route incoming messages to the right team, flag messages that read as angry or urgent so a person sees them first, draft a reply from your help content for an agent to check and send, and point out customers who have gone quiet after a problem. Each of these works from data you already hold, such as past tickets and order history. Keep the decision with a person until you have measured how often the AI gets it right, and start with whichever one takes the most repetitive work off your team.

How we’d approach a support chatbot

Our AI chatbot development page covers how these builds are planned and run. If what you need is a chatbot for your website built from your own content, TailGPT is the website chatbot product Tailwebs makes.

02 FAQ

Questions, answered

Will a chatbot replace my support team?
No. A chatbot takes the routine questions so your team can spend its time on complaints, exceptions and complex issues, which still need people. It works best as a partner to the team: a smooth handover to a person, staff trained to work with it, and unanswered questions logged so the team can fill the gaps.
Should a chatbot tell customers it is a bot?
Yes. Say so in the first message, and show how to reach a person at any point. A bot that is open about what it can do sets the right expectation; one that seems to be a person and then gets an answer wrong costs you the customer's trust. Naming it clearly also prepares the customer for the handover.
Can the same chatbot answer on the website and on WhatsApp?
It can, if the answering part is built separately from the channel: one set of content and rules, with the website widget and WhatsApp as two ways in. WhatsApp has its own account set-up, message rules and charges, which our guide to WhatsApp Business API costs and setup covers.
How long does it take to see whether a support chatbot is working?
Give it a few weeks of real traffic on one channel or topic before you judge it. Read a sample of conversations every week, track the questions it couldn't answer and how many customers came back about the same issue, and compare satisfaction with conversations handled by people over the same weeks.
Have business queries?
Chat with us