Lesson 2.4: Checking the output: spotting hallucinations

Learn why models state false things fluently, which claims carry the most risk, and a practical routine for checking facts, citations, numbers and summaries.

By the end of this lesson you will be able to explain why a language model can state something false in a confident voice, recognise the kinds of claim most likely to be wrong, check a reply with a short routine that matches your effort to the stakes, and write prompts that make errors easier to catch. You will see with numbers why a reply with many checkable facts is rarely all correct, how to test a citation in under two minutes, and how to sort the claims in a reply so that you spend your checking time where it matters. You should have read Lessons 1.1 and 2.2, since the first explained why fluent is not the same as correct, and the second showed how grounding in your own material reduces errors. No technical background is needed.

Everyone who uses an assistant regularly meets the moment: a reply that reads beautifully, cites a study you cannot find, quotes a figure that turns out to be wrong. The word for this is hallucination: output that is fluent and plausible and false. Treat it as a fact of the tool, like a spell-checker that sometimes accepts a wrong word. You do not stop using the tool. You learn where it fails and build checking into the work.

Why it happens

A model writes the continuation that is most plausible given the text so far. Plausible and true overlap a great deal, which is why the tools are useful. But they are not the same thing. When the model has solid, widespread information on a point, the plausible answer is usually the true one. When it has little or conflicting information, or when you ask about something specific, rare, recent or private, the most plausible continuation is something that sounds right: a realistic name, a believable percentage, a reference with a convincing title and a real-looking author. It does not know that it is guessing, and it has no inner alarm. Three pressures push in the same direction.

The pull to answer. Models are trained to be helpful, and a direct answer feels more helpful than "I don't know". Gaps in your prompt. As Lesson 1.1 showed, every unspecified detail is filled with a guess. Agreeable framing. A leading question gets a leading answer, so a false premise in your question can be accepted and built upon.

This is why the earlier habits matter: giving the facts, adding a safe exit and asking for quotations all attack the causes. This lesson deals with what remains.

Predict first: you ask an assistant, "Summarise the main findings of the 2023 RBI report on digital lending fraud, with three statistics." You do not attach anything, and no search is switched on. What do you expect, and what should you do with the reply? Decide before reading on.

A model with no access to the report may produce three statistics that look right and that you cannot trace to any source, and the "report" may be partly or wholly imagined. The safe way to treat the reply is as an unverified draft. Every number has to be found in the real document before it is used. Better still, attach the actual report and ask for statistics with quoted sentences, so that the facts come from your material and not from the model's memory.

The kinds of error

Not all mistakes look alike. Learning the types helps you know what to check. Type · What it looks like · ExampleInvented facts · A specific detail with no basis · A made-up policy clause or a founder's name Fake references · Plausible citations, links or quotations · A study title and author that do not exist Wrong numbers · Figures that are inaccurate or inconsistent · A percentage that does not add up Arithmetic errors · Mistakes in calculations, especially multi-step · A total that is off by a few per cent Distorted summaries · Additions, omissions or changed emphasis · A summary that includes a claim the source never made Out-of-date information · True once, no longer true · A rule that has since changed Faulty reasoning · A chain of steps that sounds right and is not · A conclusion that does not follow Going along with a false premise · Accepting an error in your question · Answering about an event that did not happen ### Where the risk is highest

Risk varies sharply by task. A useful rule: the more specific, rare, recent or checkable the claim, the more likely it is wrong, and the less the model is working from text you supplied. Higher risk · Lower riskExact figures, dates, names · General explanations of well-known ideas Citations, links, quotations · Rewriting or restructuring text you supplied Recent events or changes in rules · Brainstorming ideas, where wrong is harmless Niche facts about a small company or a person · Formatting, tone, grammar Multi-step calculations · Translating common phrases Anything about your private situation · Summarising a short text you pasted Even in the lower-risk column, mistakes happen, and a summary of a long text can still add or drop a point. But the table tells you where to start looking.

Red flags

Some features of a reply should make you pause.

Very specific details with no source: "62.4 per cent of borrowers in 2022". Round or suspiciously neat numbers that tell a clean story. A reference you cannot find: a title, author or report number that a quick search does not confirm. A quote with no location, or one that sounds oddly polished. Confidence about something obscure, such as a minor regulation or a small firm's history. An answer that matches your hopes too well. If the reply says exactly what you wanted to hear, check harder. Internal inconsistency: figures that differ between two paragraphs, or a total that is not the sum of its parts.

None of these proves an error. Each is a reason to check before you rely on the claim.

Why many facts make an error likely

Even a good model gets individual facts wrong at some small rate, and the rate adds up across a reply. Suppose each checkable claim is correct 97 times in 100. A reply with one claim is right 97 per cent of the time. A reply with ten claims is entirely right only 0.97 to the power 10, which is about 74 per cent of the time. With twenty claims it is about 54 per cent, and with forty about 30 per cent. These numbers are illustrative, and real claims are not independent, but they explain a common experience: a long, factual reply is much more likely to hold at least one error than a short one, even when each sentence feels fine.

A checking routine that fits the stakes

You cannot verify everything, and you should not try. The goal is to match effort to risk. A five-step routine does this.

1. Mark the checkable claims. Go through the reply and underline every name, number, date, quotation, citation and cause-and-effect statement. Skip the general explanation. 2. Triage by risk and stakes. For each underlined claim ask two questions: how likely is it to be wrong (specific, rare or recent claims score high), and how bad would it be if it were wrong in this document? Check the high-risk and high-stakes claims first. 3. Check against a source of truth. Use the primary document: the policy itself, the regulator's website, the company filing, your own data. A second AI reply is not a source of truth. 4. Cross-check what matters most. For a claim that carries real weight, find a second independent source. 5. Record and fix. Note what was wrong, correct the text and, if you reuse the prompt, add an instruction that prevents the error.

The time saving from triage is large. Take a reply with 30 claims, of which 6 are high-risk and 24 are low-risk. Checking a high-risk claim takes about 3 minutes, and a low-risk one a half-minute glance. Checking only the high-risk claims in depth and glancing at the rest takes 6 × 3 + 24 × 0.5 = 30 minutes. Checking all 30 in depth would take 90.

Checking particular kinds of claim

Facts, names and dates

Search for them in a source you trust. For a regulation, go to the regulator's own site. For a company fact, use the filing or the company's page. For a person, check an official profile. Be most sceptical of precise details about small or obscure subjects, because that is where the model has the least to go on.

Citations and quotations

Never trust a reference until you have opened it. A two-minute test works: take the title, author and year and search for them. Does the item exist? Does the author match? Does the year match? Does the content say what the reply claims? For a quotation, search for a distinctive phrase in the source. A reference that fails any of these should be removed. Models are known to produce citations that look right and do not exist, so a link that "looks real" is not evidence. If you need real references, ask the tool to search and to supply links, then open every one.

Numbers and calculations

Models make arithmetic mistakes, especially in long chains of steps and with large numbers or mixed units. Three habits help. Ask for the working: "Show each step and the units." Recompute the key results in a spreadsheet or calculator. And check plausibility: does a growth rate of 40 per cent a month make sense for this business, and does the total equal the sum of the parts? If a table has a "total" row, add it up. When the stakes are real, use the model to find the figures and explain the method, and do the final arithmetic yourself.

Summaries and rewrites

A summary can go wrong in two ways: it can add something that is not in the source, or omit something important. Compare it to the source with two questions. Can I point to the sentence that supports each claim in the summary? Is there a major point in the source that the summary leaves out? A useful follow-up prompt is: "List anything in your summary that is not stated in the source text, and anything important in the source that the summary leaves out." It is not infallible, but it often surfaces problems.

Reasoning

For an argument or recommendation, check the steps, not just the conclusion. Ask the model to lay out its reasoning, then look for a step that does not follow, a hidden assumption or a missing alternative. Lesson 3.1 returns to reasoning prompts.

Prompts that make errors easier to catch

You can shape a prompt so that errors surface sooner.

Supply the material and restrict the answer to it, as in Lesson 2.2. This removes the largest source of invention. Provide a safe exit: "If you are not sure or the information is not provided, say 'I don't know'." Ask for quotations or source labels so each claim can be traced. Separate fact from guess: "Mark each statement as stated in the source, inferred from the source, or general knowledge." Ask for the weak points: "List the three claims in your answer you are least sure of." This is imperfect, since the model's sense of its own uncertainty is unreliable, but it often points you to the shakiest parts. Ask for assumptions, as in Lesson 1.4. Avoid leading questions that embed a false premise. Ask "Did X happen?" and not "Why did X happen?" when you are not sure.

A word of caution about self-checking. Asking "Are you sure?" often makes the model second-guess a correct answer or defend a wrong one, and a model reviewing its own text shares its own blind spots. A second pass can catch slips of consistency, which is useful, but it is not verification. The final check must come from a source outside the model.

Worked examples

Example 1: a market note with a made-up statistic. An analyst asks for a short note on digital payments growth and receives: "According to a 2024 industry survey, 71 per cent of small merchants now accept UPI." She marks the number and the source as checkable and high-risk. A search finds no such survey, though a similar figure appears in a different year from a different body with a different value. She replaces the claim with the figure from the real source, cites it correctly and adds a line to her prompt: "Only use statistics from the sources I attach."

Example 2: a policy answer. A manager asks whether a fee can be waived and receives a confident "yes". Because the stakes are high, she follows the routine: she asks the model to quote the policy sentence, searches for it and finds the sentence says waivers apply only to accounts opened before a certain date. The model had generalised. The check took two minutes and prevented a wrong promise to a customer.

Example 3: an arithmetic slip in an analysis. An operations lead asks for the average resolution time across five teams and receives a result of 4.2 days. Asked to show the working, the model lists the five averages and divides by five, ignoring that the teams handle very different numbers of cases. A weighted average, using case counts, gives 5.1 days. The slip is not a hallucination in the sense of invented facts, but it is the same lesson: a plausible-looking number needs checking, and asking for the working made the problem visible. This is the "average of averages" error from the data course.

A policy for teams

When several people use an assistant, agree a simple rule for how much checking each kind of use needs. Stakes · Example uses · Minimum checkLow · Brainstorming, rewording your own text, internal drafts · Read it, and use judgement Medium · Internal reports, summaries for colleagues, customer emails · Verify every figure, name and date against a source High · Anything sent to a regulator, a client contract, medical, legal or financial advice, published material · Full verification, plus a second person to review Name an owner for each document: the person who signs it is responsible for its facts, whoever or whatever drafted it. Keep a short log of errors you catch, with the type and the fix. After a month the log tells you which kinds of mistakes your tool makes in your work, and which prompt changes prevented them.

Common mistakes

Treating fluent as correct. Polish is not evidence. Trusting a citation without opening it. Search for the title and author, and check the content. Checking everything equally. Triage by risk and stakes, so your effort goes where it counts. Using a second AI answer as the source of truth. Check against a primary source. Accepting the arithmetic. Recompute key figures and ask for the working. Asking "Are you sure?" as a check. It can flip a right answer. Verify externally. Embedding a false premise in the question. Ask neutral questions, and say when you are unsure that something happened. No owner and no log. Someone must be responsible, and recurring errors should lead to better prompts.

Key takeaways

A hallucination is fluent, plausible and false. It happens because the model writes what is likely, feels pressure to answer, and fills gaps with guesses. The riskiest claims are specific, rare, recent or private: exact figures, citations and quotations, recent rule changes and multi-step calculations. Errors accumulate. At 97 per cent per claim, a reply with 20 claims is entirely right only about 54 per cent of the time. Use a routine: mark checkable claims, triage by risk and stakes, check against a primary source, cross-check the critical ones and record the fixes. Ground the answer in supplied material, give a safe exit, ask for quotations, separate fact from inference and avoid leading questions. Match checking to stakes, name an owner for each document, and keep an error log.

Check yourself

1. Why can a model give a made-up citation? It writes plausible text, and a realistic title and author are a plausible continuation. It does not check that the source exists. 2. If each claim is right 95 per cent of the time, what is the chance that all of 10 claims are right? 0.95 to the power 10 = 0.599, about 60 per cent. 3. What is a quick test of a reference? Search for the title, author and year, confirm it exists and matches, and check that the content supports the claim. 4. Why is asking "Are you sure?" a weak check? The model may change a correct answer or defend an error. It shares its own blind spots. Verify against an outside source. 5. A reply has 40 claims, 8 high-risk at 3 minutes and 32 low-risk at 0.5 minutes. How long does triage take, and how long would checking all in depth take? Triage: 8 × 3 + 32 × 0.5 = 40 minutes. All in depth: 40 × 3 = 120 minutes.

Try this

This exercise tests how reliable your own tool is on a topic you know well. You need an assistant and about forty-five minutes.

1. Choose a subject you know deeply: your own team, a product you work on, a local topic or a hobby. Ask the assistant for ten specific facts about it, such as dates, figures or names, with no attachments. 2. Mark each fact as correct, partly correct or wrong, using your own knowledge or a reliable source. Count the errors and note their types. 3. Ask for three references on the subject. Check each one using the two-minute test. Note how many exist. 4. Now repeat with grounding: paste a short, accurate text on the subject and ask the same questions with "use only this text, quote the sentence, and say 'not stated' if it is not there". Count errors again. 5. Take any reply with numbers and recompute one calculation by hand or in a spreadsheet. 6. Triage one longer reply using the five-step routine, and time yourself with the triage simulator in mind. 7. Fill in this table. Test · Claims checked · Errors found · Types of errorUngrounded facts · · · References · · · Grounded facts · · · A calculation · · · A good result is a table that shows how the error rate changed once you grounded the answer, a short list of the red flags you spotted, and one sentence on the check you will always make from now on. With this lesson, Module 2 is complete: you can give examples, supply context, control format and tone, and check what comes back. Module 3 builds on these foundations with techniques for reasoning, multi-step work and reusable instructions.

Tags: Prompt Engineering Hallucinations Verification Fact-checking Reliability