Large Language Models (LLMs) can perform language-based quantitative predictions from unstructured inputs but are still prone to hallucinations and overconfident errors. Thus, it is crucial to know not only what a model predicts but also when its predictions can be trusted.
We introduce CARE-PPO, a reinforcement learning framework that connects loss prediction for uncertainty estimation with actor-critic PPO fine-tuning, enabling the joint learning of accurate numerical estimates and reliable confidence signals for language-based quantitative prediction.
CARE-PPO employs a Confidence-Aligned Reward for Estimation, defined as a function of prediction error, to provide dense error-aware feedback to the actor while guiding the critic to learn a value function aligned with prediction quality.
During inference, we repurpose the critic as a confidence estimator. Across two real-world tasks in healthcare and finance, and two Qwen-3 model scales (4B and 8B), CARE-PPO achieves strong quantitative prediction performance, producing significantly better-aligned confidence estimates through the critic than logit-based and verbalized baselines.
These gains persist under realistic out-of-distribution settings across domains, spanning linguistic and domain shifts. Finally, CARE-PPO reduces task-specific overfitting on general instruction-following prompts, consistent with the broader generalization advantages of RL fine-tuning over supervised approaches.
Blogger's Review: The innovative application of CARE-PPO in confidence estimation, combining reinforcement learning with traditional methods, demonstrates heightened predictive performance and reliability. This offers new insights into the application of language models in complex domains, especially in critical areas like healthcare and finance, further enhancing the credibility of quantitative predictions. It is hoped that more research can leverage this framework to address issues of model overfitting and uncertainty.