Microsoft reports stronger diagnostic AI benchmark results, plans hospital testing

Microsoft published new results for its AI Diagnostic Orchestrator, MAI-DxO, on October 22, reporting better performance than generalist physicians in simulated diagnostic workups. The immediate development is a research result and a plan for hospital evaluation, rather than evidence that the system already improves patient care.

In its announcement, Microsoft reported 83% diagnostic accuracy on complex cases from The New England Journal of Medicine. Physicians scored 36% when allowed reference materials and 22% without them. On cases derived from MIMIC-IV-ED emergency-department data, the reported comparison was 80% against 47%.

Those comparisons depend on the task and the reference group. They should not be read as a universal ranking of AI against every clinician, specialty or hospital team. The evaluated cases and conditions define what the numbers can support.

The SDBench project documentation explains the distinctive setup. A diagnostic agent starts with limited information and chooses questions or tests before committing to an answer. A gatekeeper reveals information, and a separate judge evaluates the diagnosis. Test costs are assessed alongside correctness, so an agent must consider how it reaches an answer.

That setup introduces an important limitation: the gatekeeper can provide a plausible synthetic result when the original case does not contain the requested information. The benchmark therefore models parts of a diagnostic encounter. It is not an observation of an AI independently examining a new patient.

Access is also qualified. The project documentation says the NEJM dataset needs a licence and the MIMIC material requires PhysioNet credentials and a data-use agreement. Its bundled synthetic cases are intended for development, not for reporting diagnostic performance.

The Verge's independent report emphasizes the need for clinical validation. Microsoft says a prospective study with Beth Israel Deaconess Medical Center will compare patient interactions with physicians' independent assessments.

Our reading is that this next stage must establish usefulness and reliability in clinical workflows. A benchmark accuracy advantage and estimated test savings alone do not answer those questions.