Large language models (LLMs) show promising results on medical question answering benchmarks, but their application in public health is limited by hallucinations and the rapid evolution of official guidance. Retrieval-Augmented Generation (RAG) mitigates these risks by grounding responses in a well-maintained corpus, yet end-to-end performance critically depends on retrieval configuration and evaluation beyond multiple-choice formats.
We extend PubHealthBench, a QA benchmark of 7,929 questions derived from UK Government public health guidance, into a retrieval-augmented setting and systematically evaluate retrieval and generation choices. We compare dense, sparse, and hybrid retrieval across multiple embedding models and corpus variants, showing that hybrid retrieval consistently improves recall and ranking quality, with chunk length and topic interacting with ranking performance.
Providing retrieved context significantly increases multiple-choice accuracy across diverse LLMs, enabling smaller open-weight models to match or outperform larger models used without retrieval, with gains primarily driven by retrieval quality and careful context selection. To assess realistic free-form answering, we introduce a rubric-based LLM-as-a-judge covering faithfulness, completeness, clarity, and factual consistency, validated against dual human annotations. Judge-human agreement is strongest for faithfulness and completeness, while factual consistency and clarity are less reliably reproduced, motivating caution when interpreting those dimensions at scale.
Overall, our results highlight retrieval as a primary lever for reliable public health QA and provide practical guidance for building and evaluating RAG systems grounded in official guidance.
Blogger's Review: This paper showcases the significance of RAG in public health by expanding the QA benchmark and systematically evaluating retrieval strategies. The effectiveness of hybrid retrieval emphasizes how to leverage retrieval to enhance LLM capabilities in a rapidly changing information environment, providing valuable insights and methodologies for future research and applications.