Large Language Models (LLMs) are increasingly utilized for mental health support, necessitating reliable evaluations of safety, empathy, and therapeutic appropriateness. However, existing mental health benchmarks are challenging to reproduce and compare due to inconsistent evaluation designs and metric definitions. We introduce CARE-MH, a unified framework for the comparable and reproducible evaluation of mental health LLMs.
Using CARE-MH, we reproduce and analyze state-of-the-art benchmarks, revealing that reproducibility strongly depends on model stability, and cross-benchmark disagreement primarily arises from differences in metric definitions. Our findings highlight the need for standardized evaluation configurations and shared metric definitions for future mental health LLM benchmarks.
Blogger's Review: CARE-MH presents a robust solution to the evaluation issues surrounding mental health LLMs, emphasizing the importance of standardization. As LLM applications in mental health grow, establishing consistent evaluation standards will be crucial for ensuring their effectiveness and safety.