We introduce VisualNoiseQA, a benchmark for active reasoning under noisy visual feedback. A text‑only LLM must solve VQA tasks by iteratively querying a fixed off‑the‑shelf visual‑language model (VLM) treated as a stochastic visual sensor. For each query we draw multiple samples and expose an empirical uncertainty signal via self‑consistency, allowing the reasoner to decide what to ask next or when to stop.
The construction is automatic and scalable: starting from diverse VQA sources and two noisy VLMs, we retain only questions where the sensor is inconsistent yet solvable by humans. We evaluate several LLM reasoners on 1,000 instances covering perception, chart understanding, and knowledge‑intensive reasoning. Results show marked differences in how models exploit uncertainty signals, making VisualNoiseQA a controlled playground for studying robust LLM reasoning in uncertain environments.
Review: This benchmark highlights the importance of calibrated uncertainty cues for LLMs and opens new avenues for multimodal active reasoning research.