Large language models (LLMs) are increasingly employed throughout the scientific discovery pipeline, from literature synthesis to hypothesis generation and experimental design. A common thread across these tasks is the need to continuously update beliefs as new evidence arrives, for which Bayesian inference offers a natural mathematical framework. Prior work has largely assumed that LLMs cannot perform principled Bayesian updates in context, resorting instead to simplified heuristics.
To challenge this assumption we devised a comprehensive evaluation framework comprising three components:
- Qualitative epistemic probes that test probabilistic coherence using philosophical‑style scenarios.
- Quantitative benchmarks where analytically tractable posteriors are known, allowing measurement of the model’s ability to recover them.
- Downstream impact analysis that examines how non‑Bayesian updates affect performance on practical tasks such as experimental design recommendations.
Results show that, when relying solely on internal language reasoning, LLMs frequently violate Bayes’ rule $P(H\mid D)=\frac{P(D\mid H)P(H)}{P(D)}$ and fail to maintain consistent belief updates. However, equipping the models with modern reasoning tools—particularly code execution capabilities—dramatically improves their performance, enabling close approximations of posteriors that are analytically intractable.
These findings suggest that current LLMs, when paired with appropriate tooling, can already perform complex belief updates within scientific workflows, substantially accelerating the efficiency and scale of AI‑powered discovery.
Review: While native probabilistic reasoning in LLMs remains imperfect, the integration of external computation transforms them into practical Bayesian inference agents, opening new avenues for rapid scientific advancement.