Recently, I wanted to write a special birthday card for my (now late) father, Bill McColl. To make it more meaningful, I asked an AI chatbot to create a short biography highlighting some of his life’s accomplishments. The AI chatbot wrote that he grew up in Los Angeles and was a great football player for the University of California in 1950’s, then went on to play for the Chicago Bears. Some of that was true—but he didn’t grow up in Los Angeles; he actually grew up in San Diego. And he didn’t attend the University of California; he attended their arch-rival, Stanford University, where he was an All-American football player. But if I didn’t know my father’s history intimately, I would never have caught these errors. The AI chatbot sounded confident and authoritative, but it was simply wrong.
This experience highlights a critical issue we’re seeing in healthcare today: the rise of artificial intelligence as a source of medical information for patients. I want to share some important insights about when AI can be helpful and when it can be dangerous.
The Growing Use of AI for Health Information
AI chatbots like ChatGPT, Google Gemini, and Anthropic Claude have become incredibly popular. Recent surveys show that 48% of consumers now use generative AI for health-related questions, and about 21% of patients report using ChatGPT specifically for health information. [1-2] Many patients use these tools 2-3 times weekly or more to determine if they need to see a doctor or to explore treatment alternatives. [2] This trend is understandable—AI is accessible 24/7, provides immediate answers, and doesn’t require an appointment.
The Problem: AI “Confabulation”
The technical term for what happened with my father’s biography is “confabulation” or “hallucination”—when AI confidently presents false information as fact. [3] This isn’t a minor glitch; it’s a fundamental characteristic of how these systems work. Studies evaluating AI chatbot responses to medical questions have found concerning results. When emergency medicine physicians evaluated AI responses to common emergency care questions, they found that 5-35% of responses contained dangerous information. [4] Another study on low back pain found that only 55.8% of AI recommendations were accurate, with 42.1% being inaccurate. [5] Even more troubling, AI chatbots often fail to provide verifiable sources for their medical advice. One study found that AI tools performed poorly on source reliability and relevance, with only 10% providing adequate references. [4] This means patients cannot easily verify whether the information they’re receiving is based on legitimate medical evidence or simply fabricated. Even worse, AI has been known to actually create references that do not actually exist.
When AI Chatbot in Healthcare Can Be Helpful
AI can help patients formulate better questions to ask their doctors and understand basic health concepts. The key difference is context and verification. In our medical practice, we occasionally use AI tools like OpenEvidence to search the medical literature. But here’s the critical distinction: OpenEvidence (www.openevidence.com), an AI chatbot currently available exclusively for physicians, provides us with citations and references to peer-reviewed medical studies that we can cross-check against our medical training and knowledge. We don’t simply accept what the AI chatbot tells us—we verify it against established medical evidence and our clinical experience.
The Risk for Patients
Patients typically don’t have the medical background to recognize when AI-generated health information doesn’t sound right. A recent analysis found that while AI chatbots can provide moderately accurate information, the accuracy varies significantly depending on the topic, and the reliability is often inadequate. [5] When AI makes mistakes about medical conditions, treatments, or when to seek emergency care, patients may not recognize the error, just as I wouldn’t have known about my father’s incorrect biography if I hadn’t known the truth. Studies have documented specific examples of dangerous AI advice, such as recommending starting CPR without checking for a pulse, or providing incomplete information about when to seek urgent or emergent care. [4] About one-third of patients who use ChatGPT for health information report requesting referrals or changing medications based solely on what the AI told them—often without consulting their physician first. [2]
How Patients Should Use AI Safely
If you choose to use AI for health information, here are essential guidelines:
- Never use AI as a substitute for medical advice. AI should only be a starting point for questions to discuss with your doctor.
- Always verify AI information with your physician. Studies show that most patients who use AI for health information do eventually consult their doctors (67.5%), which is the right approach. [2]
- Be especially cautious about emergency situations. AI has shown particularly poor performance in advising when to seek emergency care. [4]
- Don’t make treatment changes based on AI alone. Changing medications, requesting specific tests, or altering your treatment plan should only be done in consultation with your healthcare provider.
- Remember that AI cannot examine you. Medical diagnosis requires physical examination, medical history, and often diagnostic testing—none of which AI can provide.
The Doctor’s Perspective
Even as physicians, when we use AI tools, we maintain a critical eye. We check references, compare recommendations against clinical guidelines, and apply our medical training to evaluate whether the information makes sense. We understand that AI is a tool, not a replacement for medical expertise. The same principle applies to patients: AI can be a useful tool for learning about health topics, but it must be used with appropriate skepticism and always verified with qualified medical professionals.
A Recent Reminder
Just recently, I had another experience that reinforced this lesson. I asked an AI chatbot why our local football team, the 49ers, was seeded seventh in the NFL playoffs while the Rams were seeded sixth. Both teams had the same record of 12-5. The AI confidently explained that it was because the Rams had beaten the 49ers more times than vice versa during the season. But I knew that each team had won one of their two games against the other. Once again, the AI sounded authoritative and legitimate, but the information was completely wrong. If one team had won both games, that would have been the deciding factor, but AI was not aware that the teams each won one game. But I did. The reason the Rams were seeded above the 49ers was that they had a better conference record than the 49ers.
If AI can get basic, verifiable facts wrong about football—a topic with clear, publicly available statistics—imagine how much more problematic it can be with complex medical information where the stakes are your health and potentially your life. The bottom line: AI is a powerful technology that will continue to play a role in healthcare. But for now, and for the foreseeable future, it should complement—not replace—the relationship between you and your physician. When you have health questions or concerns, we’re here to provide the expertise, verification, and personalized care that AI simply cannot offer.
Please note: AI was used to help write this article.
References:
Journal of the American Medical Informatics Association: JAMIA. 2025. Brunyé TT, Mitroff SR, Elmore JG.
Journal of Medical Internet Research. 2024. Ayo-Ajibola O, Davis RJ, Lin ME, Riddell J, Kravitz RL.
Journal of the American College of Cardiology. 2024. Khera R, Oikonomou EK, Nadkarni GN, et al.
Journal of Medical Internet Research. 2024. Yau JY, Saadat S, Hsu E, et al.
Annals of the Rheumatic Diseases. 2024. Scaff SPS, Reis FJJ, Ferreira GE, Jacob MF, Saragiotto BT.
JAMA Internal Medicine. 2023. Ayers JW, Poliak A, Dredze M, et al.
Read Also: Longevity Unveiled: Exploring Peter Attia’s Guide to Healthy Aging
Milt McColl, MD, January 2026
Call The Village Doctor at (650) 851-4747 or Contact us to learn more about the practice.
