In large language models (LLMs), calibration is the primary criterion for evaluating model confidence, but relying solely on calibration is insufficient. It admits incoherent estimators, relies on the evaluation distribution, and fails to test whether the estimates can be interpreted as a consistent, underlying probability function.
What we actually need is for LLM confidence estimates to satisfy the conditions required for coherent probabilistic beliefs. We formalize these conditions along three axes (structural coherence, faithfulness, and usefulness) and operationalize them as C1 metrics.
Widely used estimators systematically violate these conditions despite appearing well-calibrated: models assign lower confidence to logically easier questions 31% of the time, and common interventions reducing RMSCE leave structural violations unchanged, suggesting that calibration is orthogonal to probabilistic validity.
RLHF and chain-of-thought improve usefulness metrics without restoring coherence. Our results show current LLM confidence estimates cannot be interpreted as coherent probabilities; our framework provides the tools to measure and close this gap.
Blogger's Review: This paper delves into the shortcomings of LLM confidence estimation and proposes a new evaluation framework that emphasizes the importance of structural coherence and faithfulness. By conducting a systematic analysis, the authors reveal the limitations of traditional calibration methods, pointing the way forward for future research.