Consistency metric selects cases for autonomous AI diagnosis
On 15 September Nature Medicine published a study of a locally deployed diagnostic AI agent. Researchers ran the same simulated clinical case five times and compared the meaning of diagnoses. Using a consistency threshold of 0.90, they allowed 272 cases of 551 cases for autonomous processing in the main configuration.
The simulations used MIRA‑v2, a set of de‑identified patient records from MIMIC‑IV. In each case the doctor‑agent interviews a patient‑agent, requests permitted tests, and issues a diagnosis with explanation, all on local hospital infrastructure. In June MIRA turned the “diagnose from description” task into a multi‑step virtual chart encounter where the agent asks questions, orders studies, and picks a plan.
For each of the 551 cases the team made five independent runs to capture variability. They defined ConsistencyDx as the agreement among the five diagnoses; it outperformed the model’s internal word probability (AUC 0.860 vs 0.747). When ConsistencyDx was at least 0.90, 272 cases went to autonomous flow, and 269 of those matched the benchmark label (98.9%); the remaining 279 cases were sent for physician review.
A stress test that removed the patient‑history prompt dropped accuracy from 90.6% to 70.2% and lowered ConsistencyDx, while internal probabilities and linguistic confidence features stayed high. Five runs required roughly five times more tokens than a single run. On the external VivaBench set, ConsistencyDx remained the best metric; at a threshold of 0.85, 32.0% of cases were kept for autonomous processing with an accuracy of 89.9%. The optimal threshold must be recalibrated for each deployment because it depends on model, generation settings, number of repeats, and the method used to compare diagnosis meaning.
🔗 Read original →
PubMed Central (PMC)
Towards autonomous medical artificial intelligence agents
Large language models (LLMs) show great potential for clinical decision-making, yet most applications remain narrow, task-specific chat tools rather than systems integrated into clinical workflows1,2. However, building physician copilots will ...
September 17, 2026 13
