Routing signals of modern vision Transformers—expert gates, attention‑residual weights and halting scores—are often used to train probes that predict whether the model’s prediction is correct. The improvement of such probes is usually interpreted as evidence that routing carries error information beyond the model’s outputs. We test this inference directly: keeping the real output‑routing pairs, we redraw correctness labels from a frozen output‑only generator fitted on disjoint data, making routing uninformative by construction.
Under this exact‑null label setting, a width‑matched MLP still reports a routing gain in 51.3% (308/600) of confidence‑only evaluations, whereas a linear comparison reports none. Fixing each training trajectory and selecting checkpoints by validation log‑loss instead of validation accuracy eliminates the detections (50/120 → 0/120, and an independently implemented probe 83/120 → 0/120), identifying accuracy‑based checkpoint selection as the cause; across all output views the raw detection rate drops from 27.5% (528/1920) to zero.
The repaired comparison is not sensitive: in two matched settings it fails to detect an implanted signal of about $$0.005\ \text{nats}$$ in 0/20 replicates, while a conditional permutation test based on an estimated routing law detects it in 11/20 and 10/20 replicates and rarely rejects under the null.
On real correctness labels, the conditional analysis yields model‑relative evidence in five DeiT attention‑residual families; in four it persists under two specified variants of the conditional law, and no family passes an additional noise criterion.
Fitting a better probe and testing for incremental information are distinct problems, each requiring its own validation.
Review