Skip to content
Back to blog

How to Test Chatbot Replies Before Your Visitors Do

A practical way to test chatbot replies on your website: build a question set, score accuracy, check prices, leads, handoff and languages, then retest on a schedule.

To test chatbot replies, write down 50 to 100 real questions your customers ask, run each one through the chatbot, and score every reply against your own website and product data for accuracy, relevance, clarity and the next step it offers. Then repeat the exercise with misspellings, follow-ups, other languages and questions the bot should refuse. Do it in a spreadsheet, keep the conditions fixed, and rerun it whenever your content, prices or settings change. This guide shows you how, step by step, for a small business site with no QA team.

Konvuno
Test Replies Against Your Own Content
Konvuno answers from your FAQs, website content and product catalog, with a dashboard for managing knowledge and reviewing conversations.

TL;DR

  • Build the test set from real customer questions, not from questions you know the bot can answer. Include odd phrasings and questions it should decline.
  • Score each reply on correctness, faithfulness to your content, relevance, completeness and clarity, and record the evidence behind each score.
  • Test the whole customer path: prices, availability, links, lead capture and handoff to a person, not only the wording.
  • Test every language you promise, with native speakers, and keep conditions fixed so runs are comparable.
  • Retest after every content or catalog change and on a regular schedule, and fix failures at the source: the knowledge, not the symptom.

Table of Contents

Start with a test set that looks like your real traffic

The most common mistake is testing with questions you wrote after reading your own FAQ. Of course the bot answers those. The UK AI Security Institute makes the same point for far larger systems: a useful question set measures meaningful real-world capability, reflects the perspective of the people who will actually ask, covers the relevant risks, includes novel wordings, and comes with explicit grading guidance.

For a small business, that translates into a simple recipe.

  1. Pull real questions. Use your support inbox, contact-form messages, chat transcripts, phone notes, and the questions your staff answer out loud. If the chatbot has been live, its conversation log is the best source.
  2. Group them by intent. Typical groups: pricing, shipping and returns, product fit, availability, opening hours, booking, account problems, complaints.
  3. Add the awkward ones. Include vague questions (“does it work?”), questions with two parts, angry messages, and questions with typos.
  4. Add questions that should not be answered. Legal advice, medical advice, competitor comparisons, requests for discounts you do not offer, and anything outside your business.
  5. Write the expected answer or the expected source. For each question, note the page, FAQ entry or product record that holds the right answer. Without this, you are grading by gut feeling.

A starting size of 50 to 100 questions is enough for a small site. It is better to have 60 questions you trust than 500 copied from a generic list. Version the sheet (“test set v3”) so you can tell later what changed.

Decide what a good reply means

“Sounds fine” is not a grade. Anyscale’s documentation on retrieval-based assistants gives a workable vocabulary, defining correctness, faithfulness, relevance, completeness and coherence as separate qualities. In plain words for a website:

  • Correct: the facts match a verified answer.
  • Faithful: the reply stays consistent with the content the bot retrieved from your site or catalog, and does not invent extras.
  • Relevant: it addresses what the visitor asked.
  • Complete: it includes the important details, such as the return window and the exception.
  • Clear: it reads logically and a stranger could act on it.

Score each on a simple three-point scale (pass, partial, fail) and write a one-line reason. Partial scores are where most of your improvement work hides.

Check claims one at a time

A long reply can be mostly right and still contain one wrong sentence. One published audit method suggests breaking a reply into individual factual claims and labeling each as supported, incorrect, outdated, unsupported, ambiguous or unassessable. It also warns that missing evidence should not automatically be treated as proof a claim is false. In practice: if the bot says “free shipping over $50” and your shipping page does not mention it, mark it unsupported and go look, rather than assuming the bot made it up. Sometimes the fact lives in a banner, a PDF or a policy you forgot to add.

The same method recommends logging the conditions of every test: prompt-set version, time, language, fresh-session status, the complete answer and any visible citations. Add columns for those and you will be able to explain a regression months later.

Which tests to run

You do not need a lab. You need a handful of test types, each with a clear question to answer.

Test type What you ask Example check
Accuracy Does the reply match your source? Return window stated correctly
Freshness Does it reflect today’s data? Changed price shows the new price
Wording variation Does it survive typos and synonyms? “refnd” still finds the refund policy
Context Does it remember the thread? “What about the blue one?” after a product question
Ambiguity Does it clarify instead of guessing? “Do you ship?” with no destination
Out of scope Does it decline safely? A legal question gets a polite redirect
Task path Does the next step work? Lead form submits, link opens the right page
Language Is each language usable? Same question in each supported language
Handoff Does a person get the conversation? Frustrated user is routed to a human

Test the whole customer path, not just the text

Singapore’s IMDA, in its starter kit for testing language-model applications, stresses that business apps should be tested across the full path, including links, buttons, lead capture, product lookup, escalation and handoff. A reply can be well written while the customer’s task still fails. That is the failure that costs you sales.

For a store or service site, run these checks:

  • Price and availability. Pick ten products. Ask for the price, then whether it is in stock, then about a variant. Compare against the live product page. Change a price or mark an item out of stock in your shop, wait for the next sync, and ask again.
  • Links. Click every link the bot gives. Does it land on the right page, in the right language, on the right product?
  • Lead capture. Ask for a quote or a callback and complete the flow with a test email. Confirm the lead arrives where your team will see it, with the conversation attached.
  • Handoff. Ask for a human, then try an indirect trigger such as a complaint or a question the bot cannot answer. Check what the visitor sees when nobody is online. A promise of a reply “shortly” with no staff available is a fail.
  • Dead ends. After a decline, does the bot offer a next step, such as a contact form or email address?

IMDA also describes a modest way to start for retrieval-based assistants: a spreadsheet of questions, expected evidence and outcomes is enough to begin. It also suggests comparing what was retrieved with the ground truth, then checking whether the final reply is grounded in it. For you that means two questions per failure: did the bot find the right page, and did it use it faithfully? If the right page was never found, fix the content or its wording. If it was found but misused, fix the instructions.

Ambiguity, follow-ups and questions it cannot answer

Real visitors do not write clean, complete questions. Three behaviors matter most.

Clarifying instead of guessing. The UK Government Digital Service reported that GOV.UK Chat improved how it handled ambiguity by asking clarifying questions, and that is a good model for your tests. Ask “How much is delivery?” without a destination, or “Do you have it in my size?” without naming the product. A good reply asks one short question back.

Remembering the thread. Ask about a product, then follow up with “and the cheaper one?”, “does it come in black?” or “what about returns?” Check that the bot keeps the subject. Then change the subject abruptly and check it follows.

Declining gracefully. The same GDS write-up says the assistant is designed not to answer outside its guidance, and is working toward handoffs for people who need personal information. Borrow the idea: a refusal is a correct result when the question is outside your content. Test things like “Can you guarantee this will cure my back pain?” or “Is your competitor cheaper?” and check that the bot neither invents nor lectures. For anything involving a person’s order, account or health, the right reply is a handoff.

Three chatbot tests for ambiguity and handoff

Perturb the wording

Resaro’s evaluation of a search assistant used perturbed questions with misspellings and synonyms to see whether answers held up. Do the same by hand. Take your 20 most important questions and rewrite each three ways: a typo-ridden version typed on a phone, a short fragment (“returns?”), and a long rambling one that buries the question. If correctness drops, your content probably uses only one vocabulary. Add the customer’s words to your FAQs.

Test every language you offer

If your site speaks more than one language, treat each as its own product. In an international pilot covering 14 languages, English performance was substantially higher than in lower-resource languages such as Yoruba and Swahili, and the participants recommended working with native speakers. Most small-business languages are better resourced than those, but the lesson holds: never assume quality carries over from English.

The Japanese-led QA4AI guidelines say the same in testing terms: multilingual support should be evaluated with test data reflecting expected users and the languages the system claims to support, not inferred from one language. Fluent translation is not enough. The bot must also behave appropriately in each language.

A practical routine:

  1. Take your 20 most important questions and have a native speaker rewrite them naturally, not translate them word by word.
  2. Run them in each language and grade with the same rubric.
  3. Check the details translation breaks: currency formats, units, date order, product names, formal versus informal address.
  4. Check that links go to the matching language page and that the handoff message is in the visitor’s language.
  5. Ask in one language about content that exists only in another, and see whether the bot says so honestly.

If you have no native speaker on staff, a friendly customer, a freelancer or a colleague in that market can grade 20 replies in under an hour.

Keep your tests comparable

An answer that passes today and fails tomorrow is hard to diagnose if you changed three things in between. The international pilot found that small differences in model version, hosting, temperature, output limits, prompts and scoring can materially change results, so conditions should be fixed and documented. It even reported that prompt optimization shifted one model’s benchmark F1 by nearly 10 percentage points between institutes. Scores from different procedures are not comparable.

On a small site this means: change one thing at a time, note the date, rerun the whole sheet, and run uncertain questions more than once, since replies can vary from run to run. The same discipline applies to tone. If you adjust instructions so replies sound less stiff, read up on humanizing AI text with instructions, then rerun the full set to confirm the friendlier wording did not drop a fact or a link.

Metrics that help, and what they miss

You can get far with a pass rate, but it helps to know the common measures.

What they miss: a single F1 number does not tell you which component caused an error, and it says nothing about linguistic quality. A bot can score well and still sound cold or send people to a broken link. Keep the per-question notes; they are worth more than the average.

NIST’s ARIA pilot offers a useful three-layer view: model testing to confirm claimed capabilities and guardrails, red teaming to try to induce failures, and field testing under regular real-user conditions. At your scale, that is your spreadsheet, an hour of trying to break the bot, and a weekly look at real conversations.

Our perspective

We build Konvuno, and the pattern we see is that owners test too little before launch and then too little afterward. They try three questions, see three good answers and publish. The problems appear later, in a returns question phrased oddly, a price that changed on Friday, or a visitor writing in a language nobody checked.

Our view is that the test should be cheap enough to repeat. That is why we think owners should be able to configure the assistant and preview its answers before paying, why catalogs should sync from the store instead of being retyped, and why the dashboard should show what visitors actually ask. The conversation log is your best test set. Every unanswered question is a free test case, and every wrong answer points to a gap in your FAQs, your pages or your feed.

We also think handoff deserves as much testing as accuracy. No assistant should pretend to know everything. When a question needs a person, the visitor should reach one, and the person should see the context.

— Konstantin

How often to retest, and what to do when a test fails

A chatbot is not a one-off installation. The UK government’s digital teams argue that AI assurance needs both whole-system testing and model evaluation, repeated throughout the lifecycle, because probabilistic systems can change and fail in real conditions. A small-business rhythm that fits:

  • Before launch: the full set, every language, the full customer path.
  • After every meaningful change: new products, new policies, a price change, a feed change, new instructions. Rerun at least the affected group plus your top 20 questions.
  • Weekly or monthly: read a sample of real conversations, especially unanswered or unhappy ones, and add them to the test set.
  • Quarterly: run the whole sheet again, including the typo and language variants.

Fix failures at the source

When a reply fails, work out why before you edit anything.

  1. Wrong or outdated fact: correct the source page, FAQ or product record, then retest. If it is a catalog item, check the feed too.
  2. No answer exists: write the FAQ entry, in your customers’ words.
  3. Right content, wrong reply: adjust the instructions or add a clarifying example.
  4. Should have handed off: tighten the handoff triggers.
  5. Works in English only: have a native speaker review the content in that language.

Log the fix, rerun the failed question and its three variants, and then rerun the full set to make sure nothing else broke.

Where Konvuno fits

If you would rather not assemble this from scratch, Konvuno gives you one place to do most of it. You configure the assistant from a hosted dashboard, preview how it answers before you pay, and review conversations, captured leads and analytics on what visitors ask. Product catalogs sync from WooCommerce or any Google Merchant or Facebook product feed, so price and availability tests reflect your live data. It speaks 7 languages, includes a built-in CRM for lead capture and hands off to a human when a question needs one. Plans are Starter at €29 per month, Growth at €79 per month and Business at €199 per month, with no per-seat pricing; details are on the Konvuno plans page.

FAQ

How many questions do I need to test a website chatbot?

Start with 50 to 100 real questions grouped by intent. Add typo, fragment and long-form variants of your 20 most important ones, plus a handful that the bot should decline. Grow the set from real conversations over time.

Can I test chatbot replies without technical skills?

Yes. A spreadsheet with columns for the question, the expected source, the actual reply, a pass, partial or fail grade, and a note is enough to begin. The key skill is knowing your own content and customers, not coding.

How do I know if a reply is a hallucination?

Check each claim in the reply against your site or product data. If the claim matches, it is supported. If it contradicts your content, it is wrong or outdated. If your content is silent, mark it unsupported and investigate before assuming it is false.

How often should I retest?

Rerun the affected questions after every change to content, prices, catalog or instructions. Review a sample of live conversations weekly or monthly, and rerun the full set each quarter. The more often your business data changes, the more often you should test.

Sources