The use of Multimodal Large Language Models (MLLMs) as judges for precise and consistent evaluations is becoming an emerging paradigm across various domains. Therefore, assessing the capability and reliability of MLLM-as-a-judge systems is crucial for ensuring trustworthy evaluations. Existing judge benchmarks categorize samples by task types but fail to capture the fundamental judgment capabilities necessary for reliable evaluation.
To address this gap, we introduce M-JudgeBench, a ten-dimensional capability-oriented benchmark designed to comprehensively assess the judgment abilities of MLLMs. Our benchmark decomposes evaluation into pairwise Chain-of-Thought (CoT) comparisons, length bias avoidance, and process error detection tasks, collectively covering ten fine-grained subtasks. This design allows for diagnosing model reliability across reasoning styles, response lengths, and cross-model variations.
Systematic evaluations reveal the inherent weaknesses in current MLLM-as-a-judge systems. To mitigate this issue, we propose Judge-MCTS, a data construction framework that generates pairwise reasoning trajectories with various correctness and lengths. Utilizing Judge-MCTS, we construct an MCTS-augmented dataset and train a series of robust judge models named M-Judger. Extensive experiments demonstrate that M-Judger outperforms existing judge benchmarks as well as M-JudgeBench.
Overall, our work lays a more principled foundation for evaluating MLLM-as-a-judge through the M-JudgeBench and Judge-MCTS framework, paving the way for future research on judge model evaluation and capability-driven judge training.
Blogger's Review: The M-JudgeBench and Judge-MCTS framework proposed in this paper provides a systematic approach to evaluating multimodal judge models, particularly in addressing model reliability and assessment capabilities. This will significantly guide future research, enhancing the credibility and applicability of the entire field.