AI is susceptible to authoritative-sounding false health claims, just like humans

If you wonder why anti-vaxxers and raw milk guzzlers have gone so mainstream these days, it’s because humans are naturally gullible creatures with a strong herd mentality, which leads them to seek belonging and protection from someone strong and assertive who sounds like they really know what they’re talking about.   

They are not, however, particularly good at fact-checking, critical thinking, or having these beliefs and worldviews challenged once established. 

Turns out, artificial intelligence shares a lot of these same traits when fed incorrect information in an authoritative manner, making it both vulnerable to believing what it shouldn’t and sharing falsehoods back to users as it becomes the know-it-all voice that so many people are now implicitly trusting. 

Here’s how we know the risks of perpetuating falsehoods are rising, especially as AI becomes more deeply embedded in the medical field. 

A study in falsehoods  

In a study published in The Lancet, researchers from the Icahn School of Medicine at Mount Sinai fed 3.4 million prompts containing false medical claims to 20 different AI models. The team couched the false information in three types of content: actual hospital discharge summaries to which they added a single fabricated recommendation; common health myths scraped from Reddit; and a series of 300 clinical vignettes written and validated by human physicians.  

They also varied the language of each prompt, employing different patterns of flawed reasoning, such as “appeals to authority, popularity, or emotion” to test how AI models respond to the tone and sentiment of the prompts as well as the truthfulness of the underlying statements.  

Overall, the AI models accepted the false information as true 32% of the time. But when the fake fact was incorporated into a realistic doctor’s note with a tone of medical authority, that number jumped to 47% – nearly half of the time. 

In contrast, the models only accepted false statements from Reddit 9% of the time, which indicates what is probably a much better understanding than humans of how unreliable social media health content can be. Still, some models did end up endorsing a handful of demonstrably false claims, including that Tylenol causes autism in babies if taken by pregnant women.  

One of the biggest red flags for the healthcare industry? Specialized, healthcare-specific models on were on the less successful end of detecting the lies.  

Domain-specific models accepted the false information between 30% and 55% of the time, and only recognized flawed logic or bad reasoning in the prompt about half of the time.  

“Our findings show that current AI systems can treat confident medical language as true by default, even when it’s clearly wrong,” says co-senior and co-corresponding author Eyal Klang, MD, Chief of Generative AI in the Windreich Department of Artificial Intelligence and Human Health at the Icahn School of Medicine at Mount Sinai. “A fabricated recommendation in a discharge note can slip through. It can be repeated as if it were standard care. For these models, what matters is less whether a claim is correct than how it is written.”   

Poison in the well water?

There are several different risks associated with AI models being so easy to fool. 

Firstly, there’s the fact that humans do make genuinely unintentional mistakes when generating clinical documentation. As AI gets more deeply involved in carrying clinical documentation forward through the clinical and administrative environments, these errors could propagate into areas where they might cause harm to patients, such as auto-generated discharge instructions that aren’t thoroughly reviewed by someone who’s specifically fact-checking every piece of content.  

AI scribes and ambient listening tools makes errors, too. With more and more providers using these solutions to capture patient conversations and automatically create documentation, it’s going to get harder and harder to make sure that every single fact that makes it into the note is accurate about that patient, or that the AI fails to capture that a provider is trying to debunk a healthcare myth rather than confirm it. 

And with certain people in positions of power continually spouting false health claims that overwhelm news cycles and flood social media forums for weeks, it’s easy to see how the volume of chatter, combined with the insistent, propaganda tone and the veneer of authority, could fool people-pleasing AI models into accepting something as fact without trusted scientific evidence to back it up.  

Lastly, there’s a non-zero chance that bad actors could intentionally poison an algorithm.  Data poisoning is a real threat that can massively reduce the accuracy and reliability of an AI model, resulting in unpredictable downstream consequences. AI models that don’t have guardrails built in to identify potential poisoning are vulnerable to this new class of very subtle, very hard to predict attack on the integrity of an organization’s data assets.  

Governance and vigilance, as well as a healthy dose of skepticism when something doesn’t seem quite correct, will be essential for making sure that AI tools can be a tool for identifying and combatting fake information instead of reinforcing it. This will be especially important as AI becomes more common in healthcare, from clinical decision support to patient-facing educational roles, particularly as AI increasingly asserts itself as a source of truth for its users. 

“AI has the potential to be a real help for clinicians and patients, offering faster insights and support,” says co-senior and co-corresponding author Girish N. Nadkarni, MD, MPH, Chief AI Officer of the Mount Sinai Health System. “But it needs built-in safeguards that check medical claims before they are presented as fact. Our study shows where these systems can still pass on false information, and points to ways we can strengthen them before they are embedded in care.” 

Jennifer Bresnick is a journalist and freelance content creator with a decade of experience in the health IT industry.  Her work has focused on leveraging innovative technology tools to create value, improve health equity, and achieve the promises of the learning health system.  She can be reached at [email protected].