Abstract
Other-initiated self-repair (OIR) is a crucial mechanism in conversational interaction, where a recipient signals an issue in speaking, hearing, or understanding, prompting the previous speaker to resolve it. In the context of conversational agents, accurately identifying these repair initiation strategies is essential for effectively addressing communication breakdowns.
While conversation analysis has shown that OIR initiation is accompanied by verbal and non-verbal signals such as gaze shifts, facial expressions, body postures, and hand gestures, existing computational approaches mainly rely on text and audio.
This paper introduces a novel multimodal model for OIR detection and classification, incorporating a set of visual features drawn from conversation analysis. We evaluate our approach on two corpora with distinct languages and interaction settings. Results demonstrate that visual information consistently improves performance over text and audio baselines, providing insights into cross-modal feature contributions across the two corpora.
Blogger's Review: This paper highlights the potential of multimodal approaches in detecting conversational repair, underscoring the significance of non-verbal signals beyond speech in communication, and offers new insights and methodological support for future dialogue systems design.