Large language models (LLMs) achieve expert‑level diagnostic accuracy in static evaluations, where the model receives a curated vignette that already condenses the diagnostically relevant findings. Such tests measure inference over pre‑selected evidence and ignore the inquiry used to obtain it. In real clinical practice, diagnosis is an iterative process: clinicians take the patient's history (Hx) through targeted questioning, perform physical examinations, and order and interpret tests, progressively narrowing the differential until a final diagnosis (Dx) is reached.
We introduce Hx2Dx ARENA, an interactive benchmark comprising 1,000 clinical cases drawn from the New England Journal of Medicine (NEJM). Each case simulates a full encounter that proceeds from initial presentation through history taking, physical exam, and diagnostic workup, all presented as a dialogue. We evaluate both open‑source and closed‑source LLMs under multiple configurations. Performance varies considerably across models in the interactive setting, and every model loses diagnostic accuracy compared with the static setting where the complete case is provided upfront.
The primary source of the accuracy gap is the ability to ask the right questions. As questions become more targeted and help build a more complete clinical picture, diagnostic performance improves. Thus, LLMs reason well once relevant data are available but remain limited in hypothesis‑driven information gathering—knowing what to ask to know. Closing this gap is a prerequisite for safely integrating LLMs into real‑world diagnostic workflows.
Review: Hx2Dx ARENA offers a realistic testbed for assessing LLMs' conversational clinical reasoning, highlighting current limitations in proactive information acquisition and pointing to clear avenues for future improvement.