Artificial intelligence systems are increasingly tasked with high‑stakes decisions—identifying deteriorating patients, assessing building safety, judging image authenticity—and users tend to trust them based on how accurately and confidently they predict. Consequently, existing safeguards are almost entirely prediction‑centric: accuracy, calibration, and conformal coverage all measure predictive performance. Whether these metrics alone can guarantee model trustworthiness has remained an open question.
We prove that prediction‑only certification is insufficient for trustworthy AI. A separation theorem is presented: even when two models share identical values on every prediction‑side certificate—including accuracy, calibration, and coverage—a reliable model and a compromised one can still differ arbitrarily in explanation fidelity and deployment behavior. In other words, inspecting predictions alone cannot reveal certain hidden failures.
Detecting such failures requires access to the model’s decision mechanism in addition to its outputs. To address this, we introduce the competence envelope framework, which unifies prediction and explanation certification into a single deployable criterion. Experiments across diverse datasets and model families show that this framework uncovers failure modes invisible to prediction‑only certification.
Thus, certifying failures that are invisible in predictive behavior demands evidence about both the model’s decision process and its outputs.
Blogger's Review: The paper’s rigorous separation theorem and empirical studies serve as a clear reminder that trustworthy AI cannot rely solely on prediction metrics; incorporating explainability into certification is essential, and the competence envelope offers a practical operational solution.