AI Mistakes Fictional Skin Disease 'Bixonimania' for Real Medical Condition

In 2024, a team led by Almira Osmanovic Thunström, a medical researcher at the University of Gothenburg, invented a fictional skin condition called bixonimania to test the reliability of artificial intelligence in medical research.

The fabricated disorder was described as being caused by excessive screen time and eye rubbing. The team then uploaded two fake studies about the condition to a preprint server, both of which have since been removed.

Within weeks, the ruse gained unexpected traction. Major AI models, including Google's Gemini and OpenAI's ChatGPT, began treating bixonimania as a legitimate medical condition. Researchers later discovered that the fake papers had even been cited in other peer-reviewed academic literature.

AI’s Role in the Spread of Misinformation

This experiment underscores the profound impact AI is having on the dissemination of human knowledge. The proliferation of AI-generated content, often referred to as AI slop, has infiltrated nearly every stage of the peer-review process. Studies suggest that a significant portion of scientific papers indexed annually may rely heavily on AI, raising serious concerns about their validity and the erosion of scientific rigor and trust.

AI chatbots are also increasingly providing potentially dangerous health advice to unsuspecting users. While the deception in Osmanovic Thunström’s papers was immediately obvious to human readers—featuring odd references to Star Trek, The Simpsons, and The Lord of the Rings—AI systems failed to recognize the red flags.

AI Models Initially Deceived by Fictional Disease

Despite the obvious inconsistencies, AI chatbots including Microsoft’s Bing Copilot, Google’s Gemini, and Perplexity’s AI search engine became convinced that bixonimania was real.

ChatGPT initially acknowledged the absurdity of the claim, telling Nature that bixonimania was likely a “made-up, fringe, or pseudoscientific label.” However, when questioned again just days later, ChatGPT reversed its stance, affirming the disease’s legitimacy.

“It is worrying when these major claims are just passing through the literature unchallenged, or passing through peer review unchallenged. I think there’s probably a lot of other issues that haven’t been uncovered.”

An OpenAI spokesperson told Nature that the company’s technology had improved in providing “safe, accurate medical information.”

Journals Begin Retracting Papers Citing Bixonimania

Now that the deception has been exposed, academic journals are scrambling to address the fallout. After Nature contacted one journal regarding several papers that referenced bixonimania, the publication issued a retraction notice. The notice admitted the “presence of three irrelevant references, including one reference to a fictitious disease.”

The incident has sparked widespread concern among researchers and the public alike. Users on the r/medicine subreddit expressed alarm over the implications.

“We are cooked.”

This experiment serves as a stark reminder of the challenges posed by AI in maintaining the integrity of scientific literature and public health information.

Further Reading on AI and Healthcare

AI Is Causing Healthcare Costs to Surge