A harmful misinformation hallucination occurs when an AI system generates false information that poses direct risks to human health, safety, legal standing, financial well-being, or reputation. This form of AI hallucination is distinguished from other types not by its technical mechanism but by its potential consequences: the hallucinated output, if acted upon, can cause real-world harm.
Harmful misinformation hallucinations are AI-generated outputs that contain false claims capable of causing damage when trusted and acted upon by users. These hallucinations are especially dangerous in high-stakes domains such as healthcare, law, emergency response, and financial advice, where incorrect information can lead to physical harm, legal liability, or loss of life 1).
A Harvard Kennedy School framework characterizes these as a distinct form of misinformation that differs from traditional human-generated misinformation because AI lacks intent or epistemic awareness, meaning it cannot recognize or prevent the generation of harmful content 2).
OpenAI's Whisper speech-to-text model, adopted by over 30,000 medical workers and 40 health systems through the Nabla platform, was found to fabricate medical information in transcriptions. The model inserted false words, phrases, race attributions, violent rhetoric, and non-existent medical treatments into audio transcriptions, even when processing hospital recordings. One machine learning engineer discovered hallucinations in half of over 100 hours of Whisper transcriptions examined. Another researcher who analyzed 26,000 transcripts found hallucinations in almost all of them 3). This occurred despite OpenAI's own warnings that Whisper should not be used in “high-risk domains” 4).
The OECD AI Incident Monitor documented cases where Microsoft Copilot provided dangerous medical advice to users, generating recommendations that could lead to harmful health decisions if followed without professional medical consultation 5).
German journalist Martin Bernklau, who worked as a court reporter, queried Microsoft's Copilot with his own name and location. The chatbot falsely accused him of being an escapee from a psychiatric institution, a convicted child abuser, and a conman preying on widowers. The AI had conflated Bernklau with the subjects of crimes he had covered in his reporting career 6).
In Mata v. Avianca, attorneys submitted a brief containing entirely fabricated legal citations generated by ChatGPT. The fabricated authorities, if accepted by the court, could have affected the outcome of the case and harmed the opposing party. The incident demonstrated how AI hallucinations in legal contexts can compromise the integrity of the judicial process 7).
Google's Bard chatbot incorrectly stated that the James Webb Space Telescope took the first images of an exoplanet. This factual error, made during a public product demonstration, contributed to a reported $100 billion drop in Alphabet's market capitalization, demonstrating how AI hallucinations can have significant financial consequences 8).
News bots hallucinating responses to queries about developing emergencies have been documented spreading unchecked falsehoods that undermine crisis mitigation efforts. In situations where accurate information is critical for public safety, AI-generated misinformation can directly endanger lives 9).
Google paused image generation in Gemini involving people after the model generated historically inaccurate images, including depicting Nazi-era German soldiers as people of color. While the model was attempting to apply diversity principles, the result was historically false content that generated significant controversy 10).