Natural language explanations are key to trust, transparency, and engagement in recommender systems. Human evaluation, while accurate, is costly and impractical for modern large‑scale platforms; single‑agent LLM‑based automation suffers from prompt sensitivity and inconsistent outputs.
MAREval addresses these issues with a structured four‑agent design: a planner uses a novel Chain of Debate (CoD) prompting strategy to assign roles and craft logically consistent evaluation plans; a moderator monitors the discussion to curb prompt drift; an arbitrator aggregates results from multiple evaluation rounds; and Monte Carlo sampling enhances robustness and alignment with human judgments.
Comprehensive experiments on the public TopicalChat benchmark and a proprietary e‑commerce recommendation dataset show that MAREval markedly outperforms state‑of‑the‑art single‑ and multi‑agent baselines in agreement with human annotations, while exhibiting far lower variability across repeated trials. In a large‑scale human‑annotation gate, MAREval meets production‑grade quality thresholds that prior evaluators fail to achieve. Online A/B testing further demonstrates statistically significant gains in user engagement and revenue metrics.
Overall, MAREval delivers a scalable, reliable, and human‑aligned solution for evaluating recommendation explanations, ready for real‑world deployment.
Review