A study into the responses of AI chatbots to everyday medical questions found that nearly 76% of the responses were accurate, according to investigators at Penn State University. Their research, published as a preprint in arXiv, found that while the AI chatbots could deliver useful information, the error rates remain high enough that they shouldn’t replace physicians for either diagnosing or suggesting treatments.
“Our work focuses explicitly on healthcare scenarios that the average internet user might ask AI, which is a perspective that prior research into large language models (LLMs) and healthcare hasn’t covered,” said the study’s senior author Amulya Yadav, PhD, associate professor of informatics and intelligent systems at Penn State. “We wanted to understand that if people are using LLMs like ChatGPT as a symptom health checker, like historically we’ve used Google, how accurate is the LLM in answering those queries, and how harmful could those responses be?”
The study examined a shift in how people seek healthcare information online. According to the researchers, “over half of U.S. adults consult online resources for medical advice,” a trend that is particularly prominent in younger adults, with near one-quarter of people under 30 using AI for health-related guidance.
The research was designed to evaluate AI performance in real-world everyday health communication, rather than in controlled testing environments that rely on medical licensing exams or clinical case studies. The researchers wrote that earlier studies “fail to account for the unstructured and often ambiguous nature of general-purpose everyday health inquiries.”
To conduct the study, the Penn State team organized a weeklong “Diagnose-a-thon” competition involving 34 participants, including faculty, staff, undergraduate students and graduate students. Participants submitted 212 prompts describing real or imagined health concerns from both patient and physician perspectives. They were allowed to use one of four publicly accessible AI models: ChatGPT-4o, ChatGPT-3.5, Gemini-1.5 Pro or Llama3-8b.
“One of the strengths of our study is we’re essentially trying to replicate real-world usage of LLMs by telling participants to choose the LLM of their choice and use it as they would on a normal day,” said lead author Bonam Mingole, a doctoral candidate in information sciences and technology at Penn State. “This type of participatory research is so important for understanding how the public uses AI in their daily life.”
Nine board-certified physicians evaluated the AI-generated responses to the prompts using a six-point scale to measure validity, quality of information, understanding and reasoning, and potential harm. About 76% of responses were considered accurate overall and showed that ChatGPT-4o was the most accurate at 84.62%, while Llama3-8b had the lowest at 50%.
The results also showed significant differences across medical specialties. Obstetrics and gynecology and otolaryngology generated the strongest AI performance, with high validity scores and low harm scores. Internal medicine, neurology, and dermatology produced the weakest results, including lower validity and higher risk of harmful responses.
The responses also shined a light on a continuing problem in healthcare research that leads to disparities in care. “The lower quality LLM-generated responses for underrepresented patient populations and rare medical conditions raises concerns about the potential of LLMs to inadvertently exacerbate existing healthcare disparities,” the researchers wrote. They added that addressing these issues “requires more than technical mechanisms, it calls for a broader commitment to equity in the data collection, model development, and evaluation processes.”
How the prompts were written also influenced AI performance. Queries between 60 and 250 characters produced the most accurate responses, while very specific prompts also improved output quality. Interestingly, the physician reviewers perceived greater risk of harm in responses generated from prompts that attempted to be written from a medical professional’s perspective.
To test whether specialized medical training could improve the answer generated by AI, the investigators enhanced the LLMs using Retrieval-Augmented Generation, by training them with medical textbooks, clinical guidelines and peer-reviewed research that is typically included in medical school curriculums.
Performance of the LLMs trained in this way were mixed. The physician reviewers preferred the baseline versions of Gemini and Llama over their medically trained counterparts, while no meaningful preference emerged for the ChatGPT models.
The researchers will now look to expand their study by collecting larger and more balanced datasets of crowdsourced health prompts and looking for ways to discourage overreliance on AI-generated medical advice.
“Like it or not, people will continue to use AI for diagnosing their health problems,” said study co-author S. Shyam Sundar, PhD, a professor in the Center for Socially Responsible AI at Penn State. “By understanding their use patterns and testing the validity of AI performance, our project helps advance literacy on the best and worst uses of AI for medical advice.”
The post AI Chatbots Do Not Consistently Deliver Accurate Health Responses appeared first on Inside Precision Medicine.
, and focus area co-lead with the Institute for Data Science in Oncology at UT MD Anderson, the team developed scalable artificial intelligence (AI) frameworks to detect, profile and classify TLSs from spatial omics data and routine pathology slides.
