Multilingual large language models (LLMs) are increasingly deployed in scenarios that require visible reasoning traces, such as verification, education, and safety auditing. Conventional evaluations, however, usually measure only final‑answer agreement or overall response similarity, leaving open whether users of different languages receive comparable supporting evidence. To address this gap we introduce a trace‑level divergence framework that treats visible reasoning traces as structured sequences and decomposes agreement into structural similarity, order consistency, and holistic similarity, with pair‑specific calibration for cross‑language comparison. Experiments span three reasoning benchmarks, seven languages, and six frontier LLMs. The most common baseline—embedding the whole response and computing cosine similarity—systematically overstates agreement: $52.1\%\sim67.3\%$ of examples in the top quartile of holistic similarity are not in the top quartile of structural similarity. This dissociation remains stable across sentence encoders and step segmentations. Blind bilingual annotation studies on two English‑anchored language pairs show that human judgments align with structural similarity rather than the holistic baseline. A same‑language control reveals that divergence is language‑specific: two English traces for paraphrased versions of the same question agree more closely than an English trace and its counterparts in other languages. In sum, answer‑level and holistic evaluations can substantially overestimate the consistency of multilingual explanatory artifacts, motivating trace‑level evaluation for reliable multilingual deployment.
Review: The paper cautions against relying solely on final answers to assess multilingual model consistency and demonstrates that fine‑grained trace comparison uncovers hidden language‑specific differences, offering a more trustworthy basis for safety audits and educational uses.