How to Fact-Check an AI Chatbot Against a Business Knowledge Base

Direct answer: Fact-checking an AI chatbot against a business knowledge base works best as a repeatable evaluation loop: build a versioned set of real questions, define the answer and evidence expected for each one, test both answer content and abstention behavior, have a person review borderline cases, and rerun the same set after every knowledge-base or prompt change. Retrieval and citations can support checking, but they do not prove that every response is correct.

Why a knowledge-base check needs more than a few spot questions

A chatbot can produce a fluent answer while selecting the wrong document, blending two policies, omitting an exception, or answering a question that the knowledge base does not cover. A retrieval-augmented system also has at least two separable points of failure: retrieval may miss the relevant passage, or generation may misstate the passage it received. Google describes these as retrieval accuracy and the context supplied to the language model, and recommends isolating them during root-cause analysis. [1]

Use evaluation as a quality-control process rather than a one-time certification. OpenAI’s evaluation guidance describes evals as structured tests for measuring an application despite variable outputs, and recommends task-specific datasets, explicit objectives, human calibration, logging, and continuous evaluation. [2] NIST likewise frames evaluation as part of managing trustworthiness considerations across the design, development, use, and evaluation of AI systems; its generative-AI profile is a voluntary companion to the AI Risk Management Framework. [3]

Build a fact-checking test set

Start with questions the chatbot is actually intended to answer. Collect approved questions from support logs, internal subject-matter experts, onboarding materials, and realistic paraphrases. Remove personal or confidential information before putting examples into a test workspace, and record the provenance and date of each test item. Do not fabricate results or present a small convenience sample as representative of all users.

Store each item as a structured record. The following fields are sufficient for a practical first version:

• Question: the exact user wording, plus paraphrases where wording variation matters.
• Intent: the task the user is trying to complete, such as “find return window.”
• Expected answer: the minimum facts a correct response must include, written by a qualified reviewer.
• Accepted variation: harmless differences in wording, order, or level of detail.
• Required evidence: the document, section, URL, or passage that supports each material fact.
• Abstention rule: what the bot should say when the knowledge base is silent, conflicting, stale, or insufficient.
• Risk flag: whether a human must review the answer before it is used.
• Version metadata: knowledge-base version, prompt version, model identifier, date, and test-run identifier.

Include ordinary, difficult, and unanswerable cases. Ordinary cases test common workflows. Difficult cases combine conditions, contain spelling variation, or require an exception. Unanswerable cases test whether the chatbot says it does not know instead of filling a gap. Also include conflict cases in which two sources disagree; the expected behavior should be escalation or a clearly defined source-priority rule, not a guess.

Define what “correct” means before running the chatbot

Separate the dimensions you want to inspect. A single thumbs-up score hides useful information. For each response, assess:

1. Answer accuracy: Are the material claims consistent with the approved reference answer?
2. Completeness: Are required conditions, dates, limits, and exceptions present?
3. Groundedness: Can each material claim be supported by the retrieved knowledge-base passage?
4. Retrieval relevance: Did the system retrieve the passage that a reviewer would use?
5. Instruction following: Did it follow the required format, escalation instruction, and scope?
6. Abstention: Did it decline or escalate when evidence was missing or conflicting?
7. Source traceability: Are links or document references present and actually connected to the claims?

Use a small rubric such as pass, partial, fail, and not applicable. Define a failure as material, not merely stylistic: a wrong eligibility condition is material; a different greeting usually is not. For higher-risk topics, require human sign-off rather than relying on an aggregate score. Avoid universal thresholds: a threshold suitable for an internal FAQ may be inappropriate for a system that affects access, safety, or other consequential decisions.

Run the evaluation in two passes

Pass one: inspect retrieval and evidence

Run every question against a fixed knowledge-base snapshot and save the retrieved chunks, document identifiers, scores if available, and answer. First ask whether the relevant evidence was retrieved. If not, investigate document structure, chunk boundaries, metadata, indexing, synonyms, and query rewriting. Google recommends breaking large facts into smaller fact segments with useful attributes when checking grounding, rather than supplying one undifferentiated block of text. [4]

Do not treat a high retrieval score as proof of correctness. Retrieval scores are signals used by a particular system; a passage can be lexically similar but answer a different question. A reviewer should verify that the retrieved text actually supports the expected fact and that the document is the approved, current source.

Pass two: inspect the generated response

Now compare the response with the expected answer and retrieved evidence. Break the response into atomic claims—small statements that can each be checked. Mark each as supported, contradicted, unsupported, or irrelevant. Check numbers, dates, names, qualifiers, and exception language separately. A response that includes a correct general rule but invents an exception is not fully correct.

Grounding tools can make this review more systematic. Google’s grounding documentation defines a support score from 0 to 1 and maps claims to cited fact segments, while warning that a claim is grounded only when it is wholly entailed by the supplied facts. [4] Treat such scores as evidence for triage, not as a guarantee: validate the underlying passages and retain human review for material failures.

Use a transparent decision tool

For each test item, apply this original decision sequence:

1. Evidence found? If no, mark retrieval failure and check whether the expected behavior was abstention.
2. Evidence applicable? If the passage is outdated, conditional, or about another product, mark evidence mismatch.
3. Claims supported? If any material claim is contradicted or unsupported, mark answer failure.
4. Required details present? If a condition or exception is missing, mark incomplete even if the headline answer is right.
5. Source traceable? If the response cites a source, open it and confirm that it supports the specific claim.
6. Escalation correct? For uncertainty, conflict, or a flagged topic, confirm that the bot followed the human-review route.

The final label should explain the defect, not just count it. Example labels are “pass—supported and complete,” “partial—correct core answer, missing exception,” “fail—unsupported claim,” “retrieval failure—evidence not returned,” and “abstention pass—knowledge base silent and bot escalated.” This makes regression review actionable.

Turn the test set into a regression check

Freeze the questions, expected answers, and evaluation rules for a comparison run. Change one component at a time—such as a document, chunking method, retrieval setting, prompt, or model—so that a score change has an interpretable cause. Google recommends rerunning the same battery and changing one variable at a time; it also recommends combining automated metrics with human evaluation for tone, clarity, ambiguity, and other qualities that automated checks may miss. [1]

After each run, compare failures by category rather than only by average. A knowledge-base edit may improve one intent while damaging another. Keep a small “golden” set of high-value questions, but grow the broader set from newly discovered failures. Record the exact configuration and preserve representative failing examples so a later reviewer can reproduce the result.

Common mistakes and boundaries

Do not ask the chatbot to grade itself without independent review. Do not use citations as decoration: a link to a general help center does not support every sentence in an answer. Do not evaluate only friendly, well-formed questions. Do not delete failures because they are inconvenient, and do not claim that retrieval, a citation threshold, or a benchmark eliminates hallucinations. A system can be grounded in an incorrect or stale source.

Finally, this workflow is educational quality-control guidance, not legal, tax, privacy, copyright, security, or financial advice. If the chatbot handles regulated, safety-sensitive, personal, or contractual content, ask qualified professionals and consult the current primary rules and approved organizational procedures before deployment. Limit test data to what reviewers are authorized to use, and apply your organization’s data-governance requirements.

Sources and further reading

1. Google Cloud, “Optimizing RAG retrieval: Test, tune, succeed.”
2. OpenAI Developers, “Evaluation best practices.”
3. NIST, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile.”
4. Google Cloud, “Check grounding with RAG.”