Voice‑interactive large language models process both what is said and how it is said, yet the two channels often disagree. This paper investigates whether such disagreement degrades speech emotion recognition. Text and audio are projected into a shared valence‑arousal‑dominance space, their Euclidean distance is computed, and a two‑component Gaussian mixture converts the distance into a continuous, threshold‑free incongruence score. Human judgments validate the score. Experiments evaluate three audio‑capable models on acted, conversational and spontaneous speech datasets (eNTERFACE'05, MELD, CMU‑MOSEI) while keeping the model constant across modalities. Results show that whenever audio is present, higher incongruence raises the odds of error in every corpus, after controlling for ground‑truth emotion and feature‑space distance. Supervised classifiers trained on the same features behave oppositely, suggesting the difficulty lies in how models combine raw evidence rather than the signal itself. Text‑only effects are unstable: swapping only the transcript‑reading model flips their sign on one corpus. Rebuilding the score without an emotion lexicon reproduces the central finding, and among six simpler label‑free alternatives it is the only one that predicts error across all three corpora. Code, data tables and prompts are released.\n\nReview: The study offers a principled way to quantify multimodal incongruence and reveals a systematic weakness in current audio‑enabled LLMs when fusing textual and acoustic cues, providing valuable guidance for future model improvements.