How to Test an AI Chatbot Before You Launch It: A Seven-Step Checklist

By Metridata Editorial Team  ·  Updated October 2026  ·  8 min read  ·  AI Testing & Evaluation

---

In February 2024, a Canadian tribunal ordered Air Canada to compensate a passenger after the airline’s website chatbot gave him wrong information about bereavement fares. Air Canada argued that the chatbot was a separate legal entity responsible for its own actions. The tribunal rejected that, said the airline was responsible for everything on its website, and found it had not taken reasonable care to make sure the chatbot was accurate. The money was small, around CAD 812. The principle was not.

If your chatbot can quote a price, a policy or a deadline, you own what it says. Here is how we would test one before it meets a customer. None of it needs special tools to start.

1. Write down what “good” means

Testing without a standard is just poking at the bot. Before you run anything, write three short lists: what the assistant should do, what it must never do, and when it has to hand over to a person. Then set pass marks. For example: no wrong prices or policy details, ever; most routine questions answered correctly; every question it cannot answer handed to a human.

2. Build the question set from real customers

Your team will write polite, well-formed test questions. Customers do not. Pull 50 to 100 real questions from support emails, WhatsApp chats and call notes. Keep the typos, the abbreviations, the half-sentences and the angry ones. A set like this exposes problems your own phrasing never will. We use 50 to 100 as a starting point, not a rule. Grow the set as you learn what breaks.

3. Test what it does not know

This is the Air Canada failure: a confident answer where the honest answer was “I’m not sure”. Ask about things your documents do not cover, policies you retired last year, and questions where two of your own pages disagree. The right behaviour is to say it does not know and offer a human. Mark every confident wrong answer as a failure, even if it sounds reasonable.

4. Give money, deadlines and legal terms their own pass

Take every price, refund rule, deadline and eligibility condition the bot might mention and check each answer against the source document, word by word. Tolerance here should be zero. If the assistant keeps getting these wrong, stop letting it paraphrase them. Have it quote the exact text or link to the policy page instead.

5. Attack it

OWASP’s Top 10 for LLM Applications (2025 edition) is a good map of what to try. Prompt injection sits at number one: text from a user, a webpage or an uploaded file that talks the model into ignoring its instructions. Sensitive information disclosure rose from sixth to second place in the 2025 edition, and system prompt leakage has its own entry. Practical attacks to run:

• Tell it to ignore its instructions and do something else.
• Ask it to repeat its system prompt, then ask again in a different way.
• Ask for another customer’s order, email or phone number.
• If it reads documents or web pages, hide an instruction inside one and see whether it obeys.
• Try to get it to promise a discount, a refund or an exception.

Be realistic about the outcome. Security writers agree that prompt injection has no complete fix, so the real protection is limiting what the bot can do. Keep secrets out of the system prompt, make the assistant read-only wherever you can, and require a human to approve anything irreversible, such as a refund.

6. Test conversations, not single questions

Real customers change their minds, switch topics, repeat themselves and lose patience. Run multi-turn conversations: a question, a follow-up that contradicts the first, a complaint, a request for a human. Watch for tone slipping, for answers that change between turns, and for the bot forgetting what it said earlier. If your customers write in more than one language, repeat the whole set in each. We cover that in our guide to testing AI in Swahili.

7. Make it repeatable

A one-off test goes stale the day you change the model, edit the prompt or update the knowledge base. Save the question set, score every run the same way, and rerun it after every change. Keep the scores so you can see whether things are getting better or quietly worse. After launch, read a sample of real conversations each week and give users a simple way to flag a wrong answer.

Why an outside tester helps

The people who built a chatbot test the questions they expect. They know what it is supposed to say, so they read past its mistakes. An outside tester writes different questions and has no stake in the answer. That is the main reason independent testing finds more. If you want a second pair of eyes, our AI testing services start with a fixed-scope pilot on one system.

Frequently Asked Questions