This work audits evaluation artifacts and potential data leakage in augmented large language model (LLM) systems for rare disease diagnosis. Using a repository of over 60,000 cases from public and independent clinical cohorts, we reconstructed a state‑of‑the‑art agentic framework and systematically quantified bias across prevailing AI paradigms. Experiments show that candidate‑space contamination artificially inflates performance: under conventional one‑vs‑rest evaluation the cross‑dataset mean Recall@1 is 54.3%, but strict cross‑dataset isolation reduces it to 40.3% (Hedges' g = 0.94). Supervised classifiers exhibit similar spatial overfitting, with an internal mean accuracy of 42.9% dropping to 16.7% on external cohorts (Cohen's d = 2.01). Further analysis reveals that LLMs often exploit retrospective therapeutic cues in clinical narratives, and temporal misalignment in foundational knowledge bases causes performance fluctuations. Based on these findings we propose a rigorous evaluation framework that couples strict data‑isolation protocols with standardized benchmarking safeguards, ensuring future diagnostic AI for rare diseases is assessed on genuine clinical robustness.
Review: The audit highlights pervasive leakage risks in current evaluation pipelines and urges researchers to adopt cross‑dataset isolation and temporal alignment safeguards; otherwise reported high accuracies may largely reflect evaluation artifacts rather than true diagnostic reasoning.