Confidence‑based voting aggregates parallel LLM rollouts by weighting each with internal signals such as token log probabilities, and has been widely studied for single‑turn reasoning. Modern LLMs, however, increasingly act as multi‑turn search agents that retrieve external documents and append them to the context. This paper shows that confidence‑based voting transfers poorly to this multi‑turn setting, and identifies the root cause as copy inflation: when retrieved documents are directly added to the context, tokens copied from those documents receive systematically inflated log probabilities, flattening confidence scores within each question and weakening the weighted vote. To address this, the authors propose Retrieval‑Grounded Voting (RGV), which scores each rollout by the lexical overlap between its final answer and the documents it retrieved. By computing the signal outside the contaminated context, RGV avoids reliance on token probabilities and requires no additional LLM calls. Experiments on four search‑agent benchmarks and five different LLMs demonstrate that RGV consistently outperforms confidence‑based voting, achieving up to +5.4% overall accuracy improvement and +35% on minority‑correct questions where the correct answer appears in only 1‑2 of 8 rollouts.
Blogger's Review: The paper pinpoints a fundamental flaw of confidence voting in multi‑turn retrieval scenarios and offers a simple yet powerful Retrieval‑Grounded Voting method. The broad experimental coverage and substantial gains make it highly relevant for developers of search‑oriented LLMs.