This work investigates how contextual priors—implemented via additive shallow fusion—reshape local decoding scores in speech retrieval tasks (MEG‑MASC and MOUS) and how this reshaping leads to a confidence‑ordering reversal. Local decoding scores and the fused top‑two margin are used as confidence measures.
Among initially incorrect predictions we observe a reversal: when the correct candidate starts near the top of the local ranking (e.g., ranks 2‑3), a larger margin makes a repair more likely; when it starts lower (ranks 21‑50), a larger margin actually reduces the repair probability. On MEG‑MASC the pooled correctness AUROC is $0.87$, but the AUROC separating repairs from residual errors drops from $0.70$ at initial ranks 2‑3 to $0.39$ at ranks 21‑50. Errors that begin beyond rank 20—inside the reversed region—constitute $46.6\%$ of all post‑fusion errors.
We propose a score‑level account: a repair must first close the initial deficit of the correct candidate, limiting its final margin, whereas a residual error can build a large margin between two incorrect candidates. A causal intervention that changes only the fusion weight shifts the reversal to deeper ranks as predicted. Under a word‑level language‑model prior, the reversal continues to move even after accuracy gains peak, indicating that a weight chosen solely for accuracy does not stabilize confidence.
A further experiment shows that reading local and prior scores separately improves selective decoding: the decoder answers on $74.5\%$ of windows (vs. $56.7\%$) while $92\%$ of output sets still contain the correct candidate.
Takeaway: After contextual fusion, confidence should retain both local and contextual evidence behind each prediction, not just the fused scores. The project website and code are publicly available for reproducibility.
Review