Insider Brief
- Researchers at Florida State University (FSU) have demonstrated that large language models (LLMs) can significantly improve diagnostic accuracy in healthcare when supplemented with lab test data.
- The study, published in npj Digital Medicine, found that GPT-4 achieved the highest performance among five models tested, with lab-enhanced inputs boosting top-1 diagnosis accuracy to 55% and lenient accuracy to 80%.
- The research supports using AI tools as clinical decision aids, potentially reducing diagnostic errors, improving patient outcomes, and lowering healthcare costs through more precise and efficient diagnoses.
PRESS RELEASE – The development of more accessible artificial intelligence (AI) models has transformed the field of health diagnoses and medicine, with AI being used for diagnostic accuracy, personalized treatment plans, interpreting medical images, streamlining operations, supporting remote patient monitoring and much more.
Researchers from the eHealth Lab at Florida State University’s School of Information have been evaluating the application of AI as a tool to aid health care providers in making more accurate patient diagnoses. The advancement has the potential to enhance treatment methods and improve patient outcomes.
Senior Author and Director for FSU’s Institute for Successful Longevity Zhe He and Visiting Assistant Professor Balu Bhasuran are among the co-authors on the multi-institutional research. The study has already garnered significant attention, with the paper being accessed more than 3,000 times since its publication in mid-March.
The paper, which was published in npj Digital Medicine, expands on FSU’s LabGenie project, a patient-engagement tool aimed at improving older adults’ understanding of lab test results.
The research team has been exploring the feasibility of using large language models (LLMs), a type of AI that learns from a large amount of text to answer questions accurately, to assist clinicians and improve differential diagnosis accuracy and efficiency. Differential diagnosis (DDx) is a critical step in clinical decision-making, helping health care providers distinguish between conditions with similar symptoms.
“The AI generated differential diagnosis is very comprehensive in covering all possible diagnoses for patients,” He said. “What this study helps show is how AI can potentially be used as a tool to help practitioners make more informed decisions for their patients.”
The study involved utilizing the LLMs to generate lists of the top one, five and ten DDx for clinicians’ evaluation. Researchers assessed the accuracy and predictive power of the LLMs and examined how incorporating lab test results impacted their diagnostic accuracy.
“What this study helps show is how AI can potentially be used as a tool to help practitioners make more informed decisions for their patients.”
– Zhe He, senior author and director for FSU’s Institute for Successful Longevity
The study tested five LLMs — GPT-4, GPT-3.5, Llama-2-70b, Claude-2 and Mixtral-8x7B — using clinical vignettes, or narrative patient-related cases, derived from 50 case reports. Their findings reveal that lab test data significantly improves diagnostic accuracy, with GPT-4 achieving the highest performance.
Specifically, GPT-4 achieved 55% top one accuracy and 60% top 10 accuracy with lab data, with lenient accuracy reaching 80%. Lab tests, including liver function, metabolic/toxicology panels and serology/immune tests, were generally interpreted correctly by the LLMs.
“When we asked the model for the top differential diagnosis, most of these models were able to produce the patient’s exact diagnosis,” Bhasuran said. “That’s very interesting because it implies that even in rare case diseases, the model is able to predict that.”
The research aims to help address well-known areas of concern often felt in health care settings from both the provider and patient perspective. Accurate diagnosis is crucial for effective patient management, directly influencing treatment decisions and overall patient outcomes. Reducing diagnostic errors helps streamline patient care, eliminating the need for excessive or repeated testing and ultimately lowering health care costs through reduced hospital stays and unnecessary procedures.
The work was supported by an Agency for Healthcare Research and Quality grant and partially supported by the University of Florida-Florida State University Clinical and Translational Science Award and the National Library of Medicine. It also included collaboration with Tampa General Hospital and coauthors from Florida State University, the National Library of Medicine, Emory University, University of South Florida and University of North Texas Health Science Center. FSU Undergraduate Research Opportunity Program (UROP) students Angelique Deville, Hailey Thompson, Maggie Awad and Yash Alva assisted in extracting key information for the case reports.
