Abstract
Large language models (LLMs) fine-tuned for forecasting can be accurate yet poorly calibrated, and their chain-of-thought (CoT) reasoning may not faithfully reflect the evidence behind a forecast. This paper explores whether internal representations provide a more direct insight into both.
We conduct experiments using Eternis-Forecaster 8B on OpenForesight, training representation-pooling probes on intermediate activations and finding significantly improved calibration. This result also holds for GLM-4.7-Flash and GLM-4.5-Air.
Next, we assess CoT faithfulness through evidence ablation and diversionary injection: removing an influential source from the prompt often alters the model's forecast while leaving the reasoning trace untouched. These probes act as lie detectors: their activations better track behavioral shifts, predicting the direction of change in 84% of cases, especially when the CoT conceals the perturbation's influence.
Finally, forced answering reveals that forecasts are largely fixed before reasoning begins: a single pre-reasoning pass recovers the committed answer and confidence, and routing questions by the spread of this pre-set answer distribution saves 30-47% of generated tokens with no loss of accuracy. Together, these results establish probing internal representations as a practical tool for calibrating, auditing, and triaging language model forecasters and reasoning models more broadly.
Blogger's Review: This paper provides valuable insights into the role of internal representations in enhancing model calibration and reasoning fidelity, highlighting a promising avenue for future research in improving the predictive capabilities of large language models.