Abstract
Large language models achieve high scores on medical knowledge assessments, yet clinical reasoning requires actively deciding what to investigate under uncertainty. We developed an agentic evaluation framework in hematologic oncology in which models must proactively request clinical data across three sequential rounds before committing to a diagnosis and treatment plan. Across 32 frontier models, the best achieved only 68% overall accuracy. Information utilization, the fraction of available data actually requested, was the strongest predictor of diagnostic accuracy (R = 0.69, P < 0.01).
Blogger's Review: This study highlights the limitations of large language models in clinical settings that require proactive information-seeking. While they perform well in knowledge assessments, their ability to make decisions under uncertainty and actively gather information needs significant improvement for practical applications.