Insider Brief
- A Microsoft AI system diagnosed complex medical cases from the New England Journal of Medicine with up to 85.5% accuracy, over four times the rate of experienced physicians.
- The AI, called MAI-DxO, simulates team-based clinical reasoning and achieves higher diagnostic accuracy while reducing medical testing costs.
- The benchmark used, SDBench, mimics real-world diagnosis by requiring sequential questioning and test ordering, revealing significant advantages of structured AI reasoning over static multiple-choice evaluations.
A new study by Microsoft AI researchers claims that an artificial intelligence system can outperform experienced physicians in one of the most demanding tests of medical reasoning, solving real diagnostic cases from the New England Journal of Medicine. The AI system, called the MAI Diagnostic Orchestrator (MAI-DxO), not only surpassed human doctors in accuracy but also delivered diagnoses more cost-effectively.
The results, published in the pre-print server arXiv and discussed in The AI Insider, raise critical questions about the future of clinical decision-making and the role of AI in medicine, especially as global healthcare systems strain under increasing patient demand and chronic workforce shortages.
The team writes in a Microsoft blog post: “As demand for healthcare continues to grow, costs are rising at an unsustainable pace, and billions of people face multiple barriers to better health – including inaccurate and delayed diagnoses. Increasingly, people are turning to digital tools for medical advice and support. Across Microsoft’s AI consumer products like Bing and Copilot, we see over 50 million health-related sessions every day. From a first-time knee-pain query to a late-night search for an urgent-care clinic, search engines and AI companions are quickly becoming the new front line in healthcare. We want to do more to help -and believe generative AI can be transformational.”
AI Diagnoses Better—And Cheaper
At the heart of the study is a benchmark developed by Microsoft AI, known as the Sequential Diagnosis Benchmark (SDBench). Instead of presenting medical cases in the simplified format of multiple-choice questions — as is common with AI evaluations — SDBench replicates the real-world diagnostic process by requiring models (or doctors) to iteratively ask questions, order tests and weigh costs before committing to a diagnosis.
Using this new benchmark, the researchers tested MAI-DxO on 304 anonymized diagnostic cases published in the NEJM’s prestigious “Case Records of the Massachusetts General Hospital” series. These are not everyday cases; instead, they are known for their diagnostic complexity and often require input from multiple medical specialists.
According to the study, a cohort of experienced physicians achieved only 20% accuracy in reaching the correct diagnosis on these cases, with an average cost of $2,963 per case. In contrast, the MAI-DxO system achieved up to 85.5% accuracy, which is over four times higher than the human baseline, and did so at a reduced cost of $2,396 in its budget-conscious mode. Even at maximum diagnostic accuracy, its cost remained lower than that of the unoptimized AI models or doctors.
Reimagining AI in Medicine
Unlike previous AI medical tools that focus on static snapshots or multiple-choice testing, MAI-DxO simulates how real doctors think: by gathering information step-by-step and refining hypotheses along the way. It does this through a simulated “virtual doctor panel,” where five AI personas debate and reason about the most informative next steps. These include roles such as Dr. Hypothesis (who ranks differential diagnoses), Dr. Test-Chooser (who recommends tests), and Dr. Stewardship (who curbs unnecessary spending).
This orchestrated approach produced what researchers describe as a new “Pareto frontier,” a point at which both diagnostic accuracy and cost are optimized better than any other method. When compared to off-the-shelf AI models like GPT-4o or Claude, MAI-DxO not only boosted accuracy across the board but often halved diagnostic costs.
The system was also model-agnostic. When layered on top of various commercial language models — ranging from OpenAI’s o3 to Google’s Gemini and Meta’s Llama — MAI-DxO consistently raised performance by about 11 percentage points. This flexibility could be important for hospitals choosing which AI tools to adopt without being locked into one vendor.
Implications for Clinical Practice
The study argues that AI’s superior diagnostic performance stems from a fundamental shift: whereas human doctors must choose between being generalists or specialists, frontier AI systems combine both traits. In one example, MAI-DxO swiftly identified a rare in-hospital alcohol poisoning case by asking a targeted question about sanitizer ingestion—an insight that eluded both human doctors and unstructured AI models, which fixated on unrelated causes.
Yet the researchers are careful not to overclaim. The test cases used in SDBench are selected for their difficulty and teaching value, not for being representative of everyday medical practice. As such, it remains to be seen how MAI-DxO performs on routine diagnoses, or how well it handles the variability, emotional complexity, and real-world constraints of clinical settings.
In the post, the team writes: “Of course, our research has important limitations. Although MAI-DxO excels at tackling the most complex diagnostic challenges, further testing is needed to assess its performance on more common, everyday presentations. Clinicians in our study worked without access to colleagues, textbooks, or even generative AI, which may feature in their normal clinical practice. This was done to enable a fair comparison to raw human performance.”
The authors also acknowledge limits around data realism, such as the use of synthetic test results when a requested exam wasn’t part of the original NEJM case. Moreover, while the system calculates test costs based on U.S. healthcare pricing, real-world variability in costs, insurance coverage, and care accessibility remains unaccounted for.
Next Steps and Broader Potential
Despite these hurdles, the study offers a promising vision for how AI could support clinical care. The Microsoft team says this is just the beginning.
“For us, this is just the first step,” the researchers write in the post. “We’re energized by the opportunities ahead. Important challenges remain before generative AI can be safely and responsibly deployed across healthcare. We need evidence drawn from real clinical environments, alongside appropriate governance and regulatory frameworks to ensure reliability, safety, and efficacy. That’s why we’re partnering with leading health organizations to rigorously test and validate these approaches—an essential step before any broader roll out.”
By helping clinicians reason through difficult cases, or potentially acting as a front-line assistant in under-resourced environments, MAI-DxO and similar systems might reduce diagnostic errors, expand access to expertise, and contain medical costs.
The researchers suggest that their benchmark, SDBench, could also be used to improve medical education by offering trainees interactive practice with complex diagnostic reasoning, aided by AI.
Going forward, Microsoft AI says it will continue working with partners to validate the system in more representative clinical settings and explore how best to deploy these capabilities, whether through clinician-facing tools or even direct-to-consumer health assistants.
