We introduce SocialArena, a benchmark that systematically defines and evaluates the social intelligence of large language models (LLMs). As LLMs are increasingly deployed in socially interactive applications such as mental‑health counseling, companion chatbots, and educational tutoring, existing evaluations remain fragmented, often focusing on isolated capabilities or single scenarios, and their reliability is largely unverified.
To address these gaps, we make three main contributions. First, we propose a unified three‑tier framework that splits social intelligence into role‑playing ability, mental‑state reasoning, and behavioral reasoning, and we design a role‑playing dialogue paradigm with explicit character thinking, enabling simultaneous assessment of these capabilities in a setting that mirrors realistic human interaction. Second, we introduce the U‑TRIMF framework to synthesize social contexts with broad coverage, preventing evaluation conclusions from being confined to specific scenarios. Third, to obtain reliable results, we invest $132.4K to recruit and train human annotators and adopt an arena‑style protocol where annotators compare model responses, provide preference labels, and offer natural‑language critiques. We hope SocialArena can offer valuable insights and guidance for future LLM improvements. Review