Large language models (LLMs) are increasingly used to support decisions under incomplete information, where deciding what to look up and when to stop matters as much as the final answer. Existing models often fall into an information trap: they pay for costly pieces of evidence that cannot change the decision, or they keep asking even when the evidence is already sufficient.
To address this, the paper proposes Decision‑Informed Value‑of‑information Acquisition (DIVA), a simple procedure that evaluates information based on its decision value. DIVA works in two steps: first, the LLM answers short numeric questions to build a small decision model that estimates the probability of a bad outcome and how each possible check would shift this probability; second, each candidate check is assigned a value‑cost ratio, and a check is performed only if the ratio is positive, stopping when no check is worth its cost.
The variant DIVA‑U additionally accounts for the LLM’s uncertainty about how strongly each check shifts the probability, by taking expectations over the distribution of possible probability changes, leading to a more robust valuation.
The method requires no additional training and can be applied to any LLM. Experiments across four domains and 20 LLMs show that DIVA‑U achieves lower regret (value lost relative to the best possible decision) than all baselines on real‑world data, mainly by stopping once evidence suffices. In synthetic scenarios where each check’s reliability is known, DIVA reduces the regret of the best baseline by 75%.
The code is released at github.com/seunghan96/diva.
Review: DIVA demonstrates that explicitly quantifying the value of information can rationally control query costs, offering a practical path to improve LLM efficiency and reliability in decision‑making tasks.