Current AI evaluation frameworks primarily focus on technical performance, including accuracy, robustness, reasoning ability, and policy compliance. While these metrics are essential, they are insufficient for systems that interact directly with users through natural language. Human-facing AI systems are increasingly employed as advisors, coaches, tutors, and companions. In these roles, their responses can influence how users reason, interpret emotions, form beliefs, calibrate trust, and make decisions. Therefore, the relevant unit of evaluation is not only the model but also the human-AI interaction.
This paper introduces psychological competence as a missing dimension in AI evaluation. We define psychological competence as the capacity of a human-facing AI system to support user cognition, emotional interpretation, and behavioral decision-making in ways that are appropriate to the user, context, and purpose of the interaction. This includes interaction properties such as framing, tone, perceived authority, responsiveness, uncertainty handling, and conversational guidance.
Existing evaluation approaches capture parts of this problem but rarely assess these psychological effects directly. Drawing on behavioral science and human-AI interaction research, we outline a conceptual framework for psychological competence and its core domains. Rather than proposing a specific benchmark, we define the construct, clarify its boundaries, and describe how it may be assessed through scenario-based probes, structured human evaluation, and model-assisted evaluation methods. We argue that psychological competence should become a core consideration for model providers, deploying organizations, researchers, and regulators concerned with the real-world effects of human-facing AI systems.
Blogger's Review: This paper highlights the importance of considering psychological dimensions in AI evaluations, pointing out that relying solely on technical performance metrics is insufficient to fully assess the impact of AI, especially in applications closely interacting with humans. Future AI evaluation frameworks should integrate measurements of psychological competence to ensure AI systems better meet user needs.