Insider Brief
- Technische Universität Dresden researchers enhanced GPT-4 with multimodal tools to create an autonomous AI agent capable of analyzing medical images, genetic data, and clinical guidelines for oncology decision support.
- The AI agent achieved 91% accuracy on simulated patient cases, with improved citation quality and reduced hallucinations due to access to 6,800 authoritative medical documents and specialized search tools.
- The study, published by EKFZ for Digital Health, underscores the agent’s promise for routine clinical use while noting the need for further validation, regulatory oversight, and human-in-the-loop integration.
PRESS RELEASE – Clinical decision-making in oncology is challenging and requires the analysis of various data types – from medical imaging and genetic information to patient records and treatment guidelines. To effectively support medical practice, AI models must be capable of processing multimodal data and have reasoning and problem-solving capabilities that resemble those of humans.
To build an autonomous AI agent for precision medicine, the researchers enhanced the large language model GPT-4 with several digital tools – including radiology report generation from MRI and CT scans, medical image analysis, prediction of genetic alterations directly from histopathology slides, and search functions across platforms such as PubMed, Google and OncoKB. To ensure that decisions were grounded in current medical knowledge, the model was given access to around 6,800 documents compiled from official oncology guidelines and clinical resources.
Autonomous AI agents successfully tested on realistic, simulated patient cases
The AI agent was evaluated on 20 real-world, simulated patient cases using a two-step process: First, the system selected appropriate tools, then it retrieved relevant medical information to guide its reasoning. The outputs were reviewed by human medical experts for accuracy, completeness, and correct citation of sources. The AI agent reached correct clinical conclusions in 91% of cases and accurately cited relevant oncology guidelines in over 75% of its responses. Importantly, the use of specialized tools and medical information retrieval significantly improved the model’s performance. As a result, so-called “hallucinations” – seemingly plausible but incorrect statements – were significantly reduced. This improvement is particularly important in the sensitive area of healthcare.
“AI tools are designed to support medical professionals, freeing up valuable time for patient care,” says Dyke Ferber, first author of the publication. “They could help in daily decision-making processes and support doctors to stay updated on the latest treatment recommendations, contributing to the identification of optimal personalized care for cancer patients.”
Potential future support in everyday clinical practice
The study serves as proof of principle that AI agents can be designed to support oncologists in everyday clinical practice. Despite the promising results, the researchers acknowledge current limitations of their study. The system was only tested on a small number of simulated cases and requires further validation. Future work will focus on integrating conversational capabilities with human feedback – so-called “human-in-the-loop” interactions – and ensuring data privacy through deployment on local servers.
“To fully leverage the potential of AI agents in the future, it will be crucial to integrate them smoothly into routine clinical practice with minimal disruption,” says Prof. Jakob N. Kather, Professor of Clinical Artificial Intelligence at EKFZ for Digital Health at TU Dresden, and oncologist at Dresden University Hospital Carl Gustav Carus. “Challenges such as interoperability with existing systems, compliance with data privacy laws, the need for regulatory approval processes as medical devices, and ensuring accountability still need to be addressed.”
Long-term, the research team envisions that similar AI agents could be adapted for use in other medical fields – provided they are equipped with appropriate tools and data. “For successful implementation of medical AI agents, medical professionals need to be well educated on how to effectively collaborate with these systems while maintaining full authority over final clinical decision-making”, Prof. Kather adds. “These agents are designed to support clinicians, but by no means to replace them.”
Overall, the study highlights the considerable potential of large language models when combined with precision oncology and search tools, establishing a solid foundation for the future use of AI-driven, personalized support systems in clinical practice.
