NeFut Logo NeFut
Admin Login

[CS.AI] Runtime Uncertainty Monitoring for LLM-Based Multi-Agent Systems

Published at: 2026-07-30 22:00 Last updated: 2026-07-30 23:39
#AI #Bayesian Network #Uncertainty

This paper investigates how multi-agent systems (MAS) based on large language models (LLMs) can support actuarial risk modeling, with a particular focus on uncertainty quantification. Actuarial workflows represent a high-stakes decision-support setting where unreliable outputs may lead to incorrect risk assessment, unfair pricing, and regulatory non-compliance. To address uncertainty introduced by the probabilistic nature of LLMs and dependencies between agents, a multi-agent framework is proposed where specialized agents perform data preparation, modeling, review, and explanation tasks under a central hub.\n\nThe main contribution is a novel approach to uncertainty propagation using token-level log-probabilities and a Bayesian Network. Importantly, log probabilities are not treated as direct probabilities of correctness or task success. Instead, length-normalized log-probability summaries are transformed into calibrated task-level confidence estimates before incorporation into the Bayesian Network.\n\nResults show that the framework reproduces baseline actuarial performance while providing additional insight into workflow stability and runtime uncertainty propagation.\n\nBlogger's Review: This paper presents an innovative approach to managing uncertainty in multi-agent systems, leveraging Bayesian networks and log probabilities, which could significantly enhance decision-making processes in high-risk environments.

Original Source: https://arxiv.org/abs/2607.25877

[h] Back to Home