In Depth
Hallucination isn't a bug in the ordinary sense. Generative models are built to produce the most plausible next piece of text, not the most true one. Most of the time plausible and true line up. When they don't, say the model has a gap in its knowledge, an ambiguous prompt, or a retrieval step that returned nothing useful, it fills the gap with something that reads correctly rather than admitting it doesn't know.
That's why hallucinations are so hard to catch in production. There's no error code, no stack trace, no dropped request. The agent returns a clean, confident answer, and unless someone knows the ground truth, nothing looks wrong. The failure is invisible until a customer acts on the bad output, and by then it's already caused harm.
The exposure scales with two things: how much the agent operates without a human checking its work, and how high the stakes of its domain are. A chatbot suggesting a recipe can hallucinate all day. A support agent quoting policy, a legal assistant citing case law, or a clinical tool summarizing a chart cannot. There, a single confident fabrication is a direct line to a claim.
What It Looks Like
In 2024, a Canadian tribunal held Air Canada liable for a refund its support chatbot had invented, a bereavement-fare policy that did not exist. The airline argued the bot was a separate entity responsible for its own statements. The tribunal disagreed: the company was responsible for everything on its website, including what its AI said. The customer got the refund, and the case became the reference point for "your agent's words are your words."
Now picture the same pattern in healthcare. An AI scribe summarizing a patient visit adds a symptom the patient never reported, or drops a medication the doctor did prescribe. The note goes into the record. The next clinician treats the patient based on it. This is no refund dispute. It's a path to a malpractice claim, with your agent in the chain of causation.
Why It Matters For AI Vendors
Hallucination is the most common AI failure and the most quotable one. It shows up in headlines, in courtrooms, and in your enterprise customer's security questionnaire. When a hallucination causes financial or physical harm, the resulting claim usually lands as misrepresentation, negligence, or professional liability. And here's the trap: the Tech E&O policy most AI vendors carry now excludes exactly this. The AI carve-outs added through 2025–2026 were written to keep "the AI said something false" claims out of traditional coverage.
So a vendor can be fully insured on paper and completely uncovered for its single most likely failure mode.