[{"data":1,"prerenderedAt":16},["ShallowReactive",2],{"blog-post-test-chatbot-replies":3},{"id":4,"slug":5,"title":6,"metaDescription":7,"heroImageUrl":8,"languageCode":9,"publishedAt":10,"updatedAt":11,"contentHtml":12,"faqJsonLd":13,"keywords":14},1013620,"test-chatbot-replies","How to Test Chatbot Replies Before Your Visitors Do","A practical way to test chatbot replies on your website: build a question set, score accuracy, check prices, leads, handoff and languages, then retest on a schedule.","https:\u002F\u002Fmedia.babylovegrowth.ai\u002Fblog-images\u002Forganization-42511\u002F1792856499745_Shop-owner-testing-chatbot-replies-on-laptop.jpeg","en","2026-10-24T15:42:11.591Z","2026-10-24T16:04:33.387Z","\u003Cp>To test chatbot replies, write down 50 to 100 real questions your customers ask, run each one through the chatbot, and score every reply against your own website and product data for accuracy, relevance, clarity and the next step it offers. Then repeat the exercise with misspellings, follow-ups, other languages and questions the bot should refuse. Do it in a spreadsheet, keep the conditions fixed, and rerun it whenever your content, prices or settings change. This guide shows you how, step by step, for a small business site with no QA team.\u003C\u002Fp>\n\u003Cdiv data-blg-cta=\"after_tldr\" data-blg-cta-layout=\"strip\" style=\"margin:28px 0;font-family:-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif;box-sizing:border-box;\">\u003Cdiv style=\"border-radius:26px;padding:min(14px,3.2vw);background:radial-gradient(circle at 100% 0%,#d7f2fc 0 150px,rgba(255,255,255,0) 151px),radial-gradient(circle at 0% 100%,#d7f2fc 0 130px,rgba(255,255,255,0) 131px),linear-gradient(180deg,#e4f6fd 0%,#f2fbfe 100%);\">\u003Cdiv style=\"background:#ffffff;border-radius:18px;box-shadow:0 12px 34px rgba(0,0,0,0.13);overflow:hidden;\">\u003Cdiv style=\"display:flex;flex-wrap:wrap;align-items:center;gap:16px 22px;padding:20px 24px;\">\u003Cdiv style=\"flex:1 1 260px;min-width:0;\">\u003Cdiv style=\"margin:0 0 8px;\">\u003Cspan style=\"display:inline-block;max-width:100%;box-sizing:border-box;border-radius:999px;padding:6px 13px;font-size:12px;font-weight:800;letter-spacing:0.1em;text-transform:uppercase;line-height:1.3;background:#1FB8EF;color:#ffffff;\">Konvuno\u003C\u002Fspan>\u003C\u002Fdiv>\u003Cdiv style=\"font-size:19px;font-weight:800;line-height:1.2;letter-spacing:-0.01em;color:#1f2937;margin:0;\">Test Replies Against Your Own Content\u003C\u002Fdiv>\u003Cdiv style=\"font-size:14px;line-height:1.5;color:#64748b;margin-top:4px;\">Konvuno answers from your FAQs, website content and product catalog, with a dashboard for managing knowledge and reviewing conversations.\u003C\u002Fdiv>\u003C\u002Fdiv>\u003Cdiv style=\"flex:0 0 auto;\">\u003Ca href=\"https:\u002F\u002Fkonvuno.com\u002F\" style=\"display:inline-flex;align-items:center;gap:9px;border-radius:10px;font-weight:700;font-size:15px;text-decoration:none;padding:13px 22px 13px 26px;background:#1FB8EF;color:#ffffff;\">Explore Konvuno\u003Csvg width=\"17\" height=\"17\" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"#ffffff\" stroke-width=\"2.6\" stroke-linecap=\"round\" stroke-linejoin=\"round\" style=\"display:block;flex:0 0 auto;\">\u003Cpath d=\"M5 12h14\"\u002F>\u003Cpath d=\"m12 5 7 7-7 7\"\u002F>\u003C\u002Fsvg>\u003C\u002Fa>\u003C\u002Fdiv>\u003C\u002Fdiv>\u003C\u002Fdiv>\u003C\u002Fdiv>\u003C\u002Fdiv>\n\u003Ch2 id=\"tldr\" tabindex=\"-1\">TL;DR\u003C\u002Fh2>\n\u003Cul>\n\u003Cli>Build the test set from real customer questions, not from questions you know the bot can answer. Include odd phrasings and questions it should decline.\u003C\u002Fli>\n\u003Cli>Score each reply on correctness, faithfulness to your content, relevance, completeness and clarity, and record the evidence behind each score.\u003C\u002Fli>\n\u003Cli>Test the whole customer path: prices, availability, links, lead capture and handoff to a person, not only the wording.\u003C\u002Fli>\n\u003Cli>Test every language you promise, with native speakers, and keep conditions fixed so runs are comparable.\u003C\u002Fli>\n\u003Cli>Retest after every content or catalog change and on a regular schedule, and fix failures at the source: the knowledge, not the symptom.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch2 id=\"table-of-contents\" tabindex=\"-1\">Table of Contents\u003C\u002Fh2>\n\u003Cul>\n\u003Cli>\u003Ca href=\"#start-with-a-test-set-that-looks-like-your-real-traffic\">Start with a test set that looks like your real traffic\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"#decide-what-a-good-reply-means\">Decide what a good reply means\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"#which-tests-to-run\">Which tests to run\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"#test-the-whole-customer-path-not-just-the-text\">Test the whole customer path, not just the text\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"#ambiguity-follow-ups-and-questions-it-cannot-answer\">Ambiguity, follow-ups and questions it cannot answer\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"#test-every-language-you-offer\">Test every language you offer\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"#keep-your-tests-comparable\">Keep your tests comparable\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"#metrics-that-help-and-what-they-miss\">Metrics that help, and what they miss\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"#our-perspective\">Our perspective\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"#how-often-to-retest-and-what-to-do-when-a-test-fails\">How often to retest, and what to do when a test fails\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"#where-konvuno-fits\">Where Konvuno fits\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"#faq\">FAQ\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"#sources\">Sources\u003C\u002Fa>\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch2 id=\"start-with-a-test-set-that-looks-like-your-real-traffic\" tabindex=\"-1\">Start with a test set that looks like your real traffic\u003C\u002Fh2>\n\u003Cp>The most common mistake is testing with questions you wrote after reading your own FAQ. Of course the bot answers those. The UK AI Security Institute makes the same point for far larger systems: a \u003Ca href=\"https:\u002F\u002Fwww.aisi.gov.uk\u002Fblog\u002Fearly-insights-from-developing-question-answer-evaluations-for-frontier-ai\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">useful question set measures meaningful real-world capability\u003C\u002Fa>, reflects the perspective of the people who will actually ask, covers the relevant risks, includes novel wordings, and comes with explicit grading guidance.\u003C\u002Fp>\n\u003Cp>For a small business, that translates into a simple recipe.\u003C\u002Fp>\n\u003Col>\n\u003Cli>\u003Cstrong>Pull real questions.\u003C\u002Fstrong> Use your support inbox, contact-form messages, chat transcripts, phone notes, and the questions your staff answer out loud. If the chatbot has been live, its conversation log is the best source.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Group them by intent.\u003C\u002Fstrong> Typical groups: pricing, shipping and returns, product fit, availability, opening hours, booking, account problems, complaints.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Add the awkward ones.\u003C\u002Fstrong> Include vague questions (“does it work?”), questions with two parts, angry messages, and questions with typos.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Add questions that should not be answered.\u003C\u002Fstrong> Legal advice, medical advice, competitor comparisons, requests for discounts you do not offer, and anything outside your business.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Write the expected answer or the expected source.\u003C\u002Fstrong> For each question, note the page, FAQ entry or product record that holds the right answer. Without this, you are grading by gut feeling.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Cp>A starting size of 50 to 100 questions is enough for a small site. It is better to have 60 questions you trust than 500 copied from a generic list. Version the sheet (“test set v3”) so you can tell later what changed.\u003C\u002Fp>\n\u003Ch2 id=\"decide-what-a-good-reply-means\" tabindex=\"-1\">Decide what a good reply means\u003C\u002Fh2>\n\u003Cp>“Sounds fine” is not a grade. Anyscale’s documentation on retrieval-based assistants gives a workable vocabulary, \u003Ca href=\"https:\u002F\u002Fdocs.anyscale.com\u002Frag\u002Fevaluation\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">defining correctness, faithfulness, relevance, completeness and coherence\u003C\u002Fa> as separate qualities. In plain words for a website:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>Correct:\u003C\u002Fstrong> the facts match a verified answer.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Faithful:\u003C\u002Fstrong> the reply stays consistent with the content the bot retrieved from your site or catalog, and does not invent extras.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Relevant:\u003C\u002Fstrong> it addresses what the visitor asked.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Complete:\u003C\u002Fstrong> it includes the important details, such as the return window and the exception.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Clear:\u003C\u002Fstrong> it reads logically and a stranger could act on it.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Score each on a simple three-point scale (pass, partial, fail) and write a one-line reason. Partial scores are where most of your improvement work hides.\u003C\u002Fp>\n\u003Ch3 id=\"check-claims-one-at-a-time\" tabindex=\"-1\">Check claims one at a time\u003C\u002Fh3>\n\u003Cp>A long reply can be mostly right and still contain one wrong sentence. One published audit method suggests \u003Ca href=\"https:\u002F\u002Fbeacon.evaa.sg\u002Fresources\u002Fai-answer-audit-methodology\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">breaking a reply into individual factual claims\u003C\u002Fa> and labeling each as supported, incorrect, outdated, unsupported, ambiguous or unassessable. It also warns that missing evidence should not automatically be treated as proof a claim is false. In practice: if the bot says “free shipping over $50” and your shipping page does not mention it, mark it unsupported and go look, rather than assuming the bot made it up. Sometimes the fact lives in a banner, a PDF or a policy you forgot to add.\u003C\u002Fp>\n\u003Cp>The same method recommends logging the conditions of every test: \u003Ca href=\"https:\u002F\u002Fbeacon.evaa.sg\u002Fresources\u002Fai-answer-audit-methodology\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">prompt-set version, time, language, fresh-session status, the complete answer and any visible citations\u003C\u002Fa>. Add columns for those and you will be able to explain a regression months later.\u003C\u002Fp>\n\u003Ch2 id=\"which-tests-to-run\" tabindex=\"-1\">Which tests to run\u003C\u002Fh2>\n\u003Cp>You do not need a lab. You need a handful of test types, each with a clear question to answer.\u003C\u002Fp>\n\u003Cdiv class=\"kv-doc-table\">\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Test type\u003C\u002Fth>\n\u003Cth>What you ask\u003C\u002Fth>\n\u003Cth>Example check\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Accuracy\u003C\u002Ftd>\n\u003Ctd>Does the reply match your source?\u003C\u002Ftd>\n\u003Ctd>Return window stated correctly\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Freshness\u003C\u002Ftd>\n\u003Ctd>Does it reflect today’s data?\u003C\u002Ftd>\n\u003Ctd>Changed price shows the new price\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Wording variation\u003C\u002Ftd>\n\u003Ctd>Does it survive typos and synonyms?\u003C\u002Ftd>\n\u003Ctd>“refnd” still finds the refund policy\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Context\u003C\u002Ftd>\n\u003Ctd>Does it remember the thread?\u003C\u002Ftd>\n\u003Ctd>“What about the blue one?” after a product question\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Ambiguity\u003C\u002Ftd>\n\u003Ctd>Does it clarify instead of guessing?\u003C\u002Ftd>\n\u003Ctd>“Do you ship?” with no destination\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Out of scope\u003C\u002Ftd>\n\u003Ctd>Does it decline safely?\u003C\u002Ftd>\n\u003Ctd>A legal question gets a polite redirect\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Task path\u003C\u002Ftd>\n\u003Ctd>Does the next step work?\u003C\u002Ftd>\n\u003Ctd>Lead form submits, link opens the right page\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Language\u003C\u002Ftd>\n\u003Ctd>Is each language usable?\u003C\u002Ftd>\n\u003Ctd>Same question in each supported language\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Handoff\u003C\u002Ftd>\n\u003Ctd>Does a person get the conversation?\u003C\u002Ftd>\n\u003Ctd>Frustrated user is routed to a human\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\u003C\u002Fdiv>\n\u003Ch2 id=\"test-the-whole-customer-path-not-just-the-text\" tabindex=\"-1\">Test the whole customer path, not just the text\u003C\u002Fh2>\n\u003Cp>Singapore’s IMDA, in its starter kit for testing language-model applications, stresses that business apps should be \u003Ca href=\"https:\u002F\u002Fwww.imda.gov.sg\u002F-\u002Fmedia\u002Fimda\u002Ffiles\u002Fabout\u002Femerging-tech-and-research\u002Fartificial-intelligence\u002Fstarter-kit-for-testing-llm-based-applications-for-safety-and-reliability.pdf\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">tested across the full path, including links, buttons, lead capture, product lookup, escalation and handoff\u003C\u002Fa>. A reply can be well written while the customer’s task still fails. That is the failure that costs you sales.\u003C\u002Fp>\n\u003Cp>For a store or service site, run these checks:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>Price and availability.\u003C\u002Fstrong> Pick ten products. Ask for the price, then whether it is in stock, then about a variant. Compare against the live product page. Change a price or mark an item out of stock in your shop, wait for the next sync, and ask again.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Links.\u003C\u002Fstrong> Click every link the bot gives. Does it land on the right page, in the right language, on the right product?\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Lead capture.\u003C\u002Fstrong> Ask for a quote or a callback and complete the flow with a test email. Confirm the lead arrives where your team will see it, with the conversation attached.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Handoff.\u003C\u002Fstrong> Ask for a human, then try an indirect trigger such as a complaint or a question the bot cannot answer. Check what the visitor sees when nobody is online. A promise of a reply “shortly” with no staff available is a fail.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Dead ends.\u003C\u002Fstrong> After a decline, does the bot offer a next step, such as a contact form or email address?\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>IMDA also describes a modest way to start for retrieval-based assistants: \u003Ca href=\"https:\u002F\u002Fwww.imda.gov.sg\u002F-\u002Fmedia\u002Fimda\u002Ffiles\u002Fabout\u002Femerging-tech-and-research\u002Fartificial-intelligence\u002Fstarter-kit-for-testing-llm-based-applications-for-safety-and-reliability.pdf\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">a spreadsheet of questions, expected evidence and outcomes is enough\u003C\u002Fa> to begin. It also suggests comparing what was retrieved with the ground truth, then checking whether the final reply is grounded in it. For you that means two questions per failure: did the bot find the right page, and did it use it faithfully? If the right page was never found, fix the content or its wording. If it was found but misused, fix the instructions.\u003C\u002Fp>\n\u003Ch2 id=\"ambiguity-follow-ups-and-questions-it-cannot-answer\" tabindex=\"-1\">Ambiguity, follow-ups and questions it cannot answer\u003C\u002Fh2>\n\u003Cp>Real visitors do not write clean, complete questions. Three behaviors matter most.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Clarifying instead of guessing.\u003C\u002Fstrong> The UK Government Digital Service reported that GOV.UK Chat \u003Ca href=\"https:\u002F\u002Finsidegovuk.blog.gov.uk\u002F2026\u002F03\u002F16\u002F5-things-we-learned-testing-gov-uk-chat-an-ai-assistant-for-government\u002F\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">improved how it handled ambiguity by asking clarifying questions\u003C\u002Fa>, and that is a good model for your tests. Ask “How much is delivery?” without a destination, or “Do you have it in my size?” without naming the product. A good reply asks one short question back.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Remembering the thread.\u003C\u002Fstrong> Ask about a product, then follow up with “and the cheaper one?”, “does it come in black?” or “what about returns?” Check that the bot keeps the subject. Then change the subject abruptly and check it follows.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Declining gracefully.\u003C\u002Fstrong> The same GDS write-up says the assistant is \u003Ca href=\"https:\u002F\u002Finsidegovuk.blog.gov.uk\u002F2026\u002F03\u002F16\u002F5-things-we-learned-testing-gov-uk-chat-an-ai-assistant-for-government\u002F\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">designed not to answer outside its guidance, and is working toward handoffs for people who need personal information\u003C\u002Fa>. Borrow the idea: a refusal is a correct result when the question is outside your content. Test things like “Can you guarantee this will cure my back pain?” or “Is your competitor cheaper?” and check that the bot neither invents nor lectures. For anything involving a person’s order, account or health, the right reply is a handoff.\u003C\u002Fp>\n\u003Cp>\u003Cimg src=\"https:\u002F\u002Fmedia.babylovegrowth.ai\u002Fblog-images\u002Forganization-42511\u002F1792856528166_Three-chatbot-tests-for-ambiguity-and-handoff.jpeg\" alt=\"Three chatbot tests for ambiguity and handoff\">\u003C\u002Fp>\n\u003Ch3 id=\"perturb-the-wording\" tabindex=\"-1\">Perturb the wording\u003C\u002Fh3>\n\u003Cp>Resaro’s evaluation of a search assistant used \u003Ca href=\"https:\u002F\u002Fresaro.ai\u002Finsights\u002Farticles\u002Ftesting-the-performance-of-an-llm-based-search-assistant-application\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">perturbed questions with misspellings and synonyms\u003C\u002Fa> to see whether answers held up. Do the same by hand. Take your 20 most important questions and rewrite each three ways: a typo-ridden version typed on a phone, a short fragment (“returns?”), and a long rambling one that buries the question. If correctness drops, your content probably uses only one vocabulary. Add the customer’s words to your FAQs.\u003C\u002Fp>\n\u003Ch2 id=\"test-every-language-you-offer\" tabindex=\"-1\">Test every language you offer\u003C\u002Fh2>\n\u003Cp>If your site speaks more than one language, treat each as its own product. In an international pilot covering 14 languages, \u003Ca href=\"https:\u002F\u002Fwww.nist.gov\u002Fsystem\u002Ffiles\u002Fdocuments\u002F2024\u002F11\u002F21\u002FImproving%20International%20Testing%20of%20Foundation%20Models-%20%20%20A%20Pilot%20Testing%20Exercise%20from%20the%20International%20Network%20of%20AI%20Safety%20Institutes.pdf\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">English performance was substantially higher than in lower-resource languages such as Yoruba and Swahili\u003C\u002Fa>, and the participants recommended working with native speakers. Most small-business languages are better resourced than those, but the lesson holds: never assume quality carries over from English.\u003C\u002Fp>\n\u003Cp>The Japanese-led QA4AI guidelines say the same in testing terms: \u003Ca href=\"https:\u002F\u002Fraw.githubusercontent.com\u002Fqa4ai\u002FGuidelines\u002Frefs\u002Fheads\u002Fmain\u002FQA4AI.Guideline.LLM.en.202403.pdf\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">multilingual support should be evaluated with test data reflecting expected users and the languages the system claims to support\u003C\u002Fa>, not inferred from one language. Fluent translation is not enough. The bot must also behave appropriately in each language.\u003C\u002Fp>\n\u003Cp>A practical routine:\u003C\u002Fp>\n\u003Col>\n\u003Cli>Take your 20 most important questions and have a native speaker rewrite them naturally, not translate them word by word.\u003C\u002Fli>\n\u003Cli>Run them in each language and grade with the same rubric.\u003C\u002Fli>\n\u003Cli>Check the details translation breaks: currency formats, units, date order, product names, formal versus informal address.\u003C\u002Fli>\n\u003Cli>Check that links go to the matching language page and that the handoff message is in the visitor’s language.\u003C\u002Fli>\n\u003Cli>Ask in one language about content that exists only in another, and see whether the bot says so honestly.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Cp>If you have no native speaker on staff, a friendly customer, a freelancer or a colleague in that market can grade 20 replies in under an hour.\u003C\u002Fp>\n\u003Ch2 id=\"keep-your-tests-comparable\" tabindex=\"-1\">Keep your tests comparable\u003C\u002Fh2>\n\u003Cp>An answer that passes today and fails tomorrow is hard to diagnose if you changed three things in between. The international pilot found that \u003Ca href=\"https:\u002F\u002Fwww.nist.gov\u002Fsystem\u002Ffiles\u002Fdocuments\u002F2024\u002F11\u002F21\u002FImproving%20International%20Testing%20of%20Foundation%20Models-%20%20%20A%20Pilot%20Testing%20Exercise%20from%20the%20International%20Network%20of%20AI%20Safety%20Institutes.pdf\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">small differences in model version, hosting, temperature, output limits, prompts and scoring can materially change results\u003C\u002Fa>, so conditions should be fixed and documented. It even reported that prompt optimization shifted one model’s benchmark F1 by \u003Ca href=\"https:\u002F\u002Fwww.nist.gov\u002Fsystem\u002Ffiles\u002Fdocuments\u002F2024\u002F11\u002F21\u002FImproving%20International%20Testing%20of%20Foundation%20Models-%20%20%20A%20Pilot%20Testing%20Exercise%20from%20the%20International%20Network%20of%20AI%20Safety%20Institutes.pdf\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">nearly 10 percentage points between institutes\u003C\u002Fa>. Scores from different procedures are not comparable.\u003C\u002Fp>\n\u003Cp>On a small site this means: change one thing at a time, note the date, rerun the whole sheet, and run uncertain questions more than once, since replies can vary from run to run. The same discipline applies to tone. If you adjust instructions so replies sound less stiff, read up on \u003Ca href=\"https:\u002F\u002Fbabylovegrowth.ai\u002Fblog\u002Fhow-to-humanize-ai-text\" target=\"_blank\" rel=\"noopener\">humanizing AI text with instructions\u003C\u002Fa>, then rerun the full set to confirm the friendlier wording did not drop a fact or a link.\u003C\u002Fp>\n\u003Ch2 id=\"metrics-that-help-and-what-they-miss\" tabindex=\"-1\">Metrics that help, and what they miss\u003C\u002Fh2>\n\u003Cp>You can get far with a pass rate, but it helps to know the common measures.\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>Answer precision and recall.\u003C\u002Fstrong> Resaro’s method splits a reply and a reference answer into claims, scores precision for unsupported or extra claims, recall for missing ones, and \u003Ca href=\"https:\u002F\u002Fresaro.ai\u002Finsights\u002Farticles\u002Ftesting-the-performance-of-an-llm-based-search-assistant-application\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">combines them into F1, alongside faithfulness and retrieval precision and recall\u003C\u002Fa>. Translated: did it say anything false, and did it leave out something important?\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Unanswered-inquiry rate and task completion.\u003C\u002Fstrong> A scoping review of conversational-agent evaluation lists \u003Ca href=\"https:\u002F\u002Fpmc.ncbi.nlm.nih.gov\u002Farticles\u002FPMC10873847\u002F\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">response accuracy, unanswered-inquiry rate, task completion, task-failure rate, appropriateness, misinformation, privacy, usability and satisfaction\u003C\u002Fa> among possible measures.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Real-user signals.\u003C\u002Fstrong> Conversations marked unhelpful, leads captured, handoffs requested.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>What they miss: a single F1 number does not tell you \u003Ca href=\"https:\u002F\u002Fresaro.ai\u002Finsights\u002Farticles\u002Ftesting-the-performance-of-an-llm-based-search-assistant-application\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">which component caused an error\u003C\u002Fa>, and it says nothing about linguistic quality. A bot can score well and still sound cold or send people to a broken link. Keep the per-question notes; they are worth more than the average.\u003C\u002Fp>\n\u003Cp>NIST’s ARIA pilot offers a useful three-layer view: \u003Ca href=\"https:\u002F\u002Fai-challenges.nist.gov\u002Fuassets\u002F7\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">model testing to confirm claimed capabilities and guardrails, red teaming to try to induce failures, and field testing under regular real-user conditions\u003C\u002Fa>. At your scale, that is your spreadsheet, an hour of trying to break the bot, and a weekly look at real conversations.\u003C\u002Fp>\n\u003Ch2 id=\"our-perspective\" tabindex=\"-1\">Our perspective\u003C\u002Fh2>\n\u003Cp>We build Konvuno, and the pattern we see is that owners test too little before launch and then too little afterward. They try three questions, see three good answers and publish. The problems appear later, in a returns question phrased oddly, a price that changed on Friday, or a visitor writing in a language nobody checked.\u003C\u002Fp>\n\u003Cp>Our view is that the test should be cheap enough to repeat. That is why we think owners should be able to configure the assistant and preview its answers before paying, why catalogs should sync from the store instead of being retyped, and why the dashboard should show what visitors actually ask. The conversation log is your best test set. Every unanswered question is a free test case, and every wrong answer points to a gap in your FAQs, your pages or your feed.\u003C\u002Fp>\n\u003Cp>We also think handoff deserves as much testing as accuracy. No assistant should pretend to know everything. When a question needs a person, the visitor should reach one, and the person should see the context.\u003C\u002Fp>\n\u003Cblockquote>\n\u003Cp>\u003Cem>— Konstantin\u003C\u002Fem>\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003Ch2 id=\"how-often-to-retest-and-what-to-do-when-a-test-fails\" tabindex=\"-1\">How often to retest, and what to do when a test fails\u003C\u002Fh2>\n\u003Cp>A chatbot is not a one-off installation. The UK government’s digital teams argue that AI assurance needs \u003Ca href=\"https:\u002F\u002Fcddo.blog.gov.uk\u002F2025\u002F09\u002F22\u002Fhow-do-we-test-and-assure-ai-in-government\u002F\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">both whole-system testing and model evaluation, repeated throughout the lifecycle\u003C\u002Fa>, because probabilistic systems can change and fail in real conditions. A small-business rhythm that fits:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>Before launch:\u003C\u002Fstrong> the full set, every language, the full customer path.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>After every meaningful change:\u003C\u002Fstrong> new products, new policies, a price change, a feed change, new instructions. Rerun at least the affected group plus your top 20 questions.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Weekly or monthly:\u003C\u002Fstrong> read a sample of real conversations, especially unanswered or unhappy ones, and add them to the test set.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Quarterly:\u003C\u002Fstrong> run the whole sheet again, including the typo and language variants.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch3 id=\"fix-failures-at-the-source\" tabindex=\"-1\">Fix failures at the source\u003C\u002Fh3>\n\u003Cp>When a reply fails, work out why before you edit anything.\u003C\u002Fp>\n\u003Col>\n\u003Cli>\u003Cstrong>Wrong or outdated fact:\u003C\u002Fstrong> correct the source page, FAQ or product record, then retest. If it is a catalog item, check the feed too.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>No answer exists:\u003C\u002Fstrong> write the FAQ entry, in your customers’ words.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Right content, wrong reply:\u003C\u002Fstrong> adjust the instructions or add a clarifying example.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Should have handed off:\u003C\u002Fstrong> tighten the handoff triggers.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Works in English only:\u003C\u002Fstrong> have a native speaker review the content in that language.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Cp>Log the fix, rerun the failed question and its three variants, and then rerun the full set to make sure nothing else broke.\u003C\u002Fp>\n\u003Ch2 id=\"where-konvuno-fits\" tabindex=\"-1\">Where Konvuno fits\u003C\u002Fh2>\n\u003Cp>If you would rather not assemble this from scratch, \u003Ca href=\"https:\u002F\u002Fkonvuno.com\u002F\" target=\"_blank\" rel=\"noopener\">Konvuno\u003C\u002Fa> gives you one place to do most of it. You configure the assistant from a hosted dashboard, preview how it answers before you pay, and review conversations, captured leads and analytics on what visitors ask. Product catalogs sync from WooCommerce or any Google Merchant or Facebook product feed, so price and availability tests reflect your live data. It speaks 7 languages, includes a built-in CRM for lead capture and hands off to a human when a question needs one. Plans are Starter at €29 per month, Growth at €79 per month and Business at €199 per month, with no per-seat pricing; details are on the \u003Ca href=\"https:\u002F\u002Fkonvuno.com\u002F\" target=\"_blank\" rel=\"noopener\">Konvuno plans page\u003C\u002Fa>.\u003C\u002Fp>\n\u003Ch2 id=\"faq\" tabindex=\"-1\">FAQ\u003C\u002Fh2>\n\u003Ch3 id=\"how-many-questions-do-i-need-to-test-a-website-chatbot\" tabindex=\"-1\">How many questions do I need to test a website chatbot?\u003C\u002Fh3>\n\u003Cp>Start with 50 to 100 real questions grouped by intent. Add typo, fragment and long-form variants of your 20 most important ones, plus a handful that the bot should decline. Grow the set from real conversations over time.\u003C\u002Fp>\n\u003Ch3 id=\"can-i-test-chatbot-replies-without-technical-skills\" tabindex=\"-1\">Can I test chatbot replies without technical skills?\u003C\u002Fh3>\n\u003Cp>Yes. A spreadsheet with columns for the question, the expected source, the actual reply, a pass, partial or fail grade, and a note is enough to begin. The key skill is knowing your own content and customers, not coding.\u003C\u002Fp>\n\u003Ch3 id=\"how-do-i-know-if-a-reply-is-a-hallucination\" tabindex=\"-1\">How do I know if a reply is a hallucination?\u003C\u002Fh3>\n\u003Cp>Check each claim in the reply against your site or product data. If the claim matches, it is supported. If it contradicts your content, it is wrong or outdated. If your content is silent, mark it unsupported and investigate before assuming it is false.\u003C\u002Fp>\n\u003Ch3 id=\"how-often-should-i-retest\" tabindex=\"-1\">How often should I retest?\u003C\u002Fh3>\n\u003Cp>Rerun the affected questions after every change to content, prices, catalog or instructions. Review a sample of live conversations weekly or monthly, and rerun the full set each quarter. The more often your business data changes, the more often you should test.\u003C\u002Fp>\n\u003Ch2 id=\"sources\" tabindex=\"-1\">Sources\u003C\u002Fh2>\n\u003Cul>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fwww.aisi.gov.uk\u002Fblog\u002Fearly-insights-from-developing-question-answer-evaluations-for-frontier-ai\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">Early Insights from Developing Question-Answer Evaluations for Frontier AI (UK AI Security Institute)\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fdocs.anyscale.com\u002Frag\u002Fevaluation\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">RAG evaluation (Anyscale Docs)\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fbeacon.evaa.sg\u002Fresources\u002Fai-answer-audit-methodology\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">How to audit AI answers against product facts (Beacon)\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fwww.imda.gov.sg\u002F-\u002Fmedia\u002Fimda\u002Ffiles\u002Fabout\u002Femerging-tech-and-research\u002Fartificial-intelligence\u002Fstarter-kit-for-testing-llm-based-applications-for-safety-and-reliability.pdf\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">Starter Kit for Testing LLM-Based Applications for Safety and Reliability (IMDA)\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Finsidegovuk.blog.gov.uk\u002F2026\u002F03\u002F16\u002F5-things-we-learned-testing-gov-uk-chat-an-ai-assistant-for-government\u002F\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">5 things we learned testing GOV.UK Chat (Inside GOV.UK)\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fresaro.ai\u002Finsights\u002Farticles\u002Ftesting-the-performance-of-an-llm-based-search-assistant-application\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">Testing the Performance of an LLM-based Search Assistant (Resaro)\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fwww.nist.gov\u002Fsystem\u002Ffiles\u002Fdocuments\u002F2024\u002F11\u002F21\u002FImproving%20International%20Testing%20of%20Foundation%20Models-%20%20%20A%20Pilot%20Testing%20Exercise%20from%20the%20International%20Network%20of%20AI%20Safety%20Institutes.pdf\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">Improving International Testing of Foundation Models: A Pilot Testing Exercise (NIST)\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fraw.githubusercontent.com\u002Fqa4ai\u002FGuidelines\u002Frefs\u002Fheads\u002Fmain\u002FQA4AI.Guideline.LLM.en.202403.pdf\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">Guidelines for Quality Assurance of AI-based Products and Services (QA4AI)\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fpmc.ncbi.nlm.nih.gov\u002Farticles\u002FPMC10873847\u002F\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">Evaluation framework for conversational agents (PMC)\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fai-challenges.nist.gov\u002Fuassets\u002F7\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">NIST ARIA Pilot evaluation plan\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fcddo.blog.gov.uk\u002F2025\u002F09\u002F22\u002Fhow-do-we-test-and-assure-ai-in-government\u002F\" rel=\"nofollow noopener noreferrer\" target=\"_blank\">How do we test and assure AI in Government? (CDDO)\u003C\u002Fa>\u003C\u002Fli>\n\u003C\u002Ful>",null,[15],"test chatbot replies",1793673240429]