The paper observes that recent work increasingly treats LLM reasoning failure taxonomies as quantitative metrics: papers report the fraction of errors that fall into a named category and use these fractions to compare models. We focus on one category, Compositional Collapse (the model notices a required sub‑part of a problem but then drops or misapplies it), and measure it on a public multilingual math benchmark covering twelve models, eleven languages, and 2,316 graded records.\n\nFirst, more than half of the failures (50.1%) provide evidence for more than one category, so the reported share depends on the order in which the classifier tests the categories. Swapping the order of two tests changes the proportion in eleven of the twelve models, and across all six possible orderings the share of Collapse ranges from 26.4 % to 76.4 %.\n\nSecond, human agreement on the label is extremely low. Native speakers of all eleven languages judged 179 transcripts, achieving κ = 0.932 on answer correctness but only κ = 0.093 on the Collapse label. A second native speaker re‑rated 142 items; for the 121 determinate pairs the two humans agree with κ = 0.659, while each agrees with the inspector at κ …\n\nThese findings demonstrate that current failure taxonomies are heavily influenced by measurement procedures and cannot serve as robust cross‑model benchmarks.\n\nReview: When using LLM error taxonomies for evaluation, researchers must account for classifier ordering and low inter‑annotator reliability, otherwise the conclusions may be misleading.