Key takeaway: Why does ChatGPT make things up? It generates the most plausible next words, not verified facts, and it has been trained and scored on tests where a guess earns points and "I don't know" earns zero. ChatGPT makes things up because guessing is what its training rewarded, so when it does not know, it fills the blank with something that sounds right.
Why does ChatGPT make things up? Short version: it was built to sound right, and then it was graded like a student who loses nothing by guessing. Nobody taught it the difference between a fact and a fluent sentence, so it does not reliably know when it has crossed from one to the other.
That is less sinister than "the AI lies to you," and in some ways worse. A liar knows the truth and hides it. ChatGPT has no private copy of the truth to hide. It has odds.
You have probably met the result. You ask for three studies on magnesium and sleep, and you get three tidy citations with authors, journals, and years. Two of them check out. The third has a real journal, a plausible title, and no existence whatsoever. The model was not trying to fool you. It was finishing the pattern you asked for, and a citation is an easy pattern to finish.
A student who never leaves a question blank
In September 2025, OpenAI published a paper with a title that sounds like a confession: "Why Language Models Hallucinate." Its authors, Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang, argue that hallucinations are not a mysterious glitch. They are a predictable statistical outcome of how these systems are trained and then evaluated.¹
The argument has two parts, and the second one is the interesting one.
The first part is about pretraining. A language model learns by predicting the next word across enormous amounts of text. That text comes with no labels saying which sentences are true. The model only ever sees fluent examples, and it has to infer the shape of language from them.²
That works beautifully for patterns. Spelling and grammar follow rules that show up millions of times, so those errors fade as models grow. Arbitrary facts are different. OpenAI's own illustration is a birthday: no pattern in the world tells you when a particular person was born.² The paper formalizes this. If a fact like a birthday appears only once in the training data, the model has very little to go on, and the authors show that its error rate on such facts should be at least roughly the share of them that appear only once.¹
Rare facts are where models guess, because there is no pattern to learn and only one example to remember.
The second part is about what happens next. After pretraining, models are tuned and then ranked on benchmarks. Most of those benchmarks score answers as right or wrong. A blank, or an "I don't know," scores the same as a wrong answer: zero.¹
Anyone who sat a multiple choice exam understands the incentive. If there is no penalty for a wrong answer, you never leave a question blank. OpenAI makes the same comparison in its summary of the paper, and adds a sharper version: asked for someone's birthday, a model that guesses a date has a 1 in 365 chance of scoring. A model that admits it does not know is guaranteed to score nothing.² Over thousands of questions, the guesser climbs the leaderboard.
When the researchers asked a widely used chatbot for the title of Kalai's own PhD dissertation, it confidently produced three different answers. None was correct. It did the same with his birthday.²
I think of this as the never blank student. Not a liar. A test taker who has learned, correctly, that confidence is free and silence is expensive.
The bill arrives in court
So what does a guess look like when it leaves the chat window?
The best known answer is a personal injury case against an airline. In Mata v. Avianca, lawyers filed a brief citing a run of court decisions, including one called Varghese v. China Southern Airlines. The cases did not exist. One of the lawyers had used ChatGPT for research, and the chatbot had supplied the opinions, quotations included. On June 22, 2023, Judge P. Kevin Castel of the Southern District of New York sanctioned the two lawyers and their firm, imposed a $5,000 penalty, and ordered them to write to each real judge who had been falsely named as the author of a fake opinion.³
That was supposed to be the cautionary tale that ended the problem. It was the opening act.
Damien Charlotin, a legal researcher, maintains a public database of court and tribunal decisions that deal with hallucinated material in filings, mostly fake or wrong citations. When I checked it on November 9, 2026, it listed 2,266 cases.⁴ Not all involve lawyers; plenty involve people representing themselves. The common thread is a confident paragraph that nobody checked.
Companies have learned the same lesson more cheaply. In 2024, a Canadian tribunal ordered Air Canada to compensate a customer, Jake Moffatt, after the airline's website chatbot said a bereavement fare could be claimed a bereavement fare retroactively. It could not. Air Canada argued that it could not be held liable for what its chatbot said. The tribunal disagreed, found that the airline had not taken reasonable care to make sure the chatbot was accurate, and awarded Moffatt $650.88 in damages plus interest and fees.⁵ A small sum, and a useful precedent: the chatbot's guess is your problem if it is on your website.
Citations are the most visible failure because they are the easiest to check. William Walters and Esther Isabelle Wilder asked ChatGPT to write short literature reviews on 42 topics and then hunted down all 636 references. Of the works cited by GPT 3.5, 55 percent did not exist. For GPT 4, the figure fell to 18 percent. Even among real works, many citations had errors in dates, volumes, or page numbers.⁶
Eighteen percent is a big improvement. It also means roughly one reference in five pointed to nothing. A research assistant with that record would be let go.
In defense of the confident machine
Now let me argue against my own headline for a minute, because "graded to guess" makes the companies sound like they forgot to fix a typo.
Guessing is also how a model is useful. A system that refuses whenever it is less than certain is safe and nearly worthless. The tension shows up in OpenAI's own numbers. In the system card for its o3 and o4 mini reasoning models, released in April 2025, the company reported that o3 hallucinated on 33 percent of questions in an internal test about people, PersonQA, roughly double the 16 percent of its predecessor o1. It also got more answers right: 59 percent versus 47 percent. OpenAI's explanation was that o3 simply makes more claims, both true and false, and that more research was needed to understand why.⁷
That is the trade in one table. The bolder model knows more and invents more.
The OpenAI paper's own example cuts the other way. On a benchmark called SimpleQA, the older o4 mini answered 24 percent of questions correctly and 75 percent wrongly, abstaining almost never. A newer GPT 5 thinking model answered 22 percent correctly, got 26 percent wrong, and declined to answer the other 52 percent.² By the usual leaderboard logic, the older model wins. By the logic of anyone who has to act on the answer, it is not close.
Which is why my headline is a little too kind. "Graded to guess" sounds like an accident of schooling. The grading is a choice, made by the industry, about what counts as a good model. The paper's proposed fix is almost embarrassingly simple: penalize confident errors more than abstentions, and give credit for appropriate uncertainty, the way some standardized tests have long used negative marking.¹ The obstacle is not mathematics. It is that nobody wants to top a leaderboard by saying "I don't know" more often than the competition.
And the models are improving. When OpenAI launched GPT 5 in August 2025, it said that with web search enabled, on prompts resembling real ChatGPT traffic, GPT 5's responses were about 45 percent less likely to contain a factual error than GPT 4o, and that its thinking mode was about 80 percent less likely to contain one than o3.⁸ Those are the company's own measurements, so read them as a direction, not a guarantee.
What happens when the question is about your body at 2 a.m.
This is a sleep and health blog, so here is the part that keeps me up.
People increasingly ask chatbots about their bodies, often at night, when no doctor is answering. A KFF poll in 2024 found that about one in six American adults, 17 percent, used AI chatbots at least once a month for health information and advice, rising to a quarter of adults under 30. Most adults, including 56 percent of people who use AI, said they were not confident the health information was accurate.⁹ People use a tool they do not trust because it is the one that is awake.
The risk at 2 a.m. is not only that the model guesses. It is that you hand it something to guess about. A team at Mount Sinai built 300 clinical scenarios, each containing a single invented detail, such as a lab test or a syndrome that does not exist, and fed them to six large language models. Depending on the model and the setup, the systems treated the fake detail as real and elaborated on it in 50 to 82 percent of cases.
A prompt instructing the model to use only validated information and flag uncertainty cut the average from about 66 percent to 44 percent. For the best performer, GPT 4o, it fell from 53 to 23 percent. Turning the randomness setting down to zero did nothing useful.¹⁰
Translate that out of the lab. You half remember a sleep supplement a friend mentioned, or misread a test result, and ask the chatbot to explain it. If your premise is wrong, the most fluent continuation is to accept it and keep going. The model is not checking your question. It is completing it.
That is why I keep repeating the same advice in different clothes. We argued earlier that a chatbot makes a poor judge of your night's sleep (https://thinknarratives.com/articles/stop-asking-a-chatbot-how-you-slept), and that passing a medical exam is not the same as handling a real appointment (https://thinknarratives.com/articles/chatgpt-passed-the-medical-exam-you-are-failing-the-appointment). Hallucination is the mechanism under both arguments. A machine that fills blanks is most dangerous exactly where you cannot see the blank.
Give it permission to say it does not know
If the model guesses because guessing pays, the practical move is to change what pays, at least inside your own conversation.
Tell it that "I don't know" is an acceptable answer. The Mount Sinai results suggest that explicit instructions to flag uncertainty reduce, though do not eliminate, invented detail.¹⁰ It costs one sentence.
Ask for sources, then open them. A citation from a chatbot is a lead, not evidence. Given the Walters and Wilder numbers, assume any reference is unconfirmed until you have seen the page yourself.⁶ Tools that search the web and link to what they found are better. You still have to click.
Do not smuggle the answer into the question. "Why does magnesium cure insomnia?" invites a confident essay. "Is there good evidence that magnesium helps with sleep?" invites an assessment. Same topic, very different odds.
Ask twice and compare. Researchers at Oxford reported in Nature in 2024 that one useful warning sign of a made up answer is inconsistency: if you sample several answers to the same question and they disagree in meaning, the model is probably confabulating.¹¹ It is the dissertation title problem again. Three different confident answers mean no real answer.
None of this is effort the chatbot will do for you, which is the uncomfortable link to an argument we made before: AI does not make you lazy, but it makes effort optional (https://thinknarratives.com/articles/ai-will-not-make-you-lazy-it-will-make-effort-optional). Checking is the effort. Skip it and you are the lawyer in the Avianca case, just with lower stakes and no judge.
The scoreboard will change before the habit does
Here is what I expect. Hallucinations will keep falling in the tests companies publish, and they will not reach zero, because some questions have no answer that a model can know. The OpenAI paper says as much: accuracy will never reach 100 percent, but a model can still learn to abstain.¹
The bigger change will be in what counts as winning. Within a few years, the serious benchmarks will penalize confident wrong answers, and "knows when to say I don't know" will become a selling point in model launches, the way battery life became one for phones. The fake case count in courtrooms will keep rising for a while anyway, because the tools are spreading faster than the habit of checking.
What won't change is the basic deal. ChatGPT will not learn what true means in the way you do. It will get better at estimating when it is likely to be wrong, and better at telling you so. The job of deciding whether to believe it stays with you, preferably before you file it with a court or swallow it with a glass of water at 2 a.m.
FAQ
Why does ChatGPT hallucinate? Because it generates text by predicting likely next words rather than looking facts up, and because training and evaluation have historically rewarded guessing over admitting uncertainty. OpenAI's 2025 research argues that standard benchmarks score "I don't know" the same as a wrong answer, so models learn to guess.
Can you stop ChatGPT from making things up? Not completely, but you can reduce it. Tell it that saying it does not know is acceptable, ask it to use web search and show links, avoid questions that contain a false premise, and check every source yourself. Asking the same question more than once and comparing the answers can also expose a made up reply.
Does ChatGPT make up sources? It can. In a 2023 study of 636 citations, 55 percent of the references produced by GPT 3.5 and 18 percent of those produced by GPT 4 did not exist, and many real references contained errors. Treat any citation as unverified until you have found the original.
Is ChatGPT getting better at accuracy? Yes, by OpenAI's own measurements. The company reported that GPT 5 makes noticeably fewer factual errors than earlier models, especially with web search or its thinking mode. Progress has not been a straight line, though: OpenAI's o3 model hallucinated more than its predecessor on one internal test, and no model is error free.
Is it safe to ask ChatGPT about sleep or health problems? It can help you understand terms or prepare questions for a doctor, but it should not be your final answer. Studies show that chatbots often accept and elaborate on false details in medical questions. Check health claims against reliable sources and talk to a clinician about symptoms that worry you or do not go away.
This article is for general information and is not medical or legal advice. For persistent sleep problems or other health concerns, speak to a doctor or another qualified health professional.
Sources
¹ Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala and Edwin Zhang, "Why Language Models Hallucinate," arXiv:2509.04664, September 2025. https://arxiv.org/abs/2509.04664
² OpenAI, "Why language models hallucinate," September 5, 2025. https://openai.com/index/why-language-models-hallucinate/
³ Mata v. Avianca, Inc., No. 1:22-cv-01461 (PKC), Opinion and Order on Sanctions, U.S. District Court for the Southern District of New York, June 22, 2023. https://law.justia.com/cases/federal/district-courts/new-york/nysdce/1:2022cv01461/575368/54/
⁴ Damien Charlotin, "AI Hallucination Cases" database, accessed November 9, 2026 (2,266 cases listed). https://www.damiencharlotin.com/hallucinations/
⁵ Moffatt v. Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal of British Columbia, February 14, 2024. https://www.canlii.org/en/bc/bccrt/doc/2024/2024bccrt149/2024bccrt149.html
⁶ William H. Walters and Esther Isabelle Wilder, "Fabrication and errors in the bibliographic citations generated by ChatGPT," Scientific Reports 13, 14045, September 2023. https://doi.org/10.1038/s41598-023-41032-5
⁷ OpenAI, "OpenAI o3 and o4-mini System Card," April 2025, section 3.3 Hallucinations. https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf
⁸ OpenAI, "Introducing GPT-5," August 7, 2025. https://openai.com/index/introducing-gpt-5/
⁹ KFF, "KFF Health Misinformation Tracking Poll: Artificial Intelligence and Health Information," August 2024 (survey fielded June 3 to 24, 2024). https://www.kff.org/public-opinion/kff-health-misinformation-tracking-poll-artificial-intelligence-and-health-information/
¹⁰ Mahmud Omar, Vera Sorin, Jeremy D. Collins and colleagues, "Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support," Communications Medicine 5, 330, August 2025. https://doi.org/10.1038/s43856-025-01021-3
¹¹ Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn and Yarin Gal, "Detecting hallucinations in large language models using semantic entropy," Nature 630, 625 to 630, June 2024. https://doi.org/10.1038/s41586-024-07421-0