NeFut Logo NeFut
Admin Login

[CS.AI] Risk Is Not Review Value: Wrong-Answer Exposure Under Bounded Review Budgets

Published at: 2026-09-11 22:00 Last updated: 2026-09-12 06:35
#AI #Machine Learning #LLM

LLM assistants often generate far more answers than humans can review. Conventional evaluations ask whether an answer is wrong, unsupported, or low‑confidence, but under a bounded review budget the key question becomes which answers to check first given limited resources. Risk alone is insufficient for prioritization: a high‑risk answer may be hard to fix, whereas a moderately risky one can often be corrected directly from available evidence. This work models review prioritization as exposure reduction and defines review value as a combination of estimated wrongness, intervention affordance, impact, and cost. Evaluation metrics include Wrong‑Answer Exposure Ratio (WAER), the fraction of wrong answers left unreviewed, and Post‑Repair Residual Exposure (PRRE), the fraction still exposed after deterministic benchmark‑supported repairs. PRRE uses repairability rules that do not reuse the affordance scores employed for ranking. Experiments on a 720‑item TAT‑QA/SciFact stress benchmark show that with a 20% review budget, review‑value ranking keeps answer‑level WAER nearly unchanged (0.605 vs. 0.600) while lowering PRRE from 0.881 to 0.716. These results indicate that trustworthy LLM evaluation should measure not only error detection but also how limited review capacity reduces exposed wrong answers.

Review

Original Source: https://arxiv.org/abs/2609.07095

[h] Back to Home