In medical multiple-choice question answering (MCQA), Retrieval-Augmented Generation (RAG) can supplement the domain knowledge of language models (LMs). However, vanilla RAG can degrade LM performance due to its indiscriminate use of retrieved documents. To address this, we propose MedJudgeRAG. Our framework represents retrieved documents as a dynamic knowledge graph (KG) composed of entities and relations.
For each option, the model judges an evidence verdict from the retrieved documents and the KG. Based on the verdict combination, the model determines a knowledge utilization strategy to reason toward the final answer. These capabilities are trained via supervised fine-tuning using structured reasoning traces generated by a teacher LM. The training employs a weighted cross-entropy loss that differentially weights the KG and reasoning segments.
Experiments on two medical MCQA benchmarks demonstrate that MedJudgeRAG consistently outperforms both vanilla RAG and parametric baselines. Furthermore, ablation analysis reveals that the dynamic KG is more effective as graph-conditioned supervision at training time than as an explicit output at inference time. Our code is available at GitHub, and the generated reasoning traces are released at Hugging Face.
Blogger's Review: The introduction of MedJudgeRAG marks a significant advancement in the field of medical multiple-choice question answering. By integrating knowledge graphs with language models, this approach not only enhances reasoning capabilities but also provides new insights for intelligent question-answering systems in the medical domain. The application of dynamic knowledge graphs further improves model performance in complex scenarios, making it a promising area for future research.