Abstract
Large language models (LLMs) are increasingly used to answer questions about political information, especially in election-adjacent settings where factual errors and ideological distortions are high-stakes. We present a reproducible measurement framework that treats hallucinations—unsupported statements in document-grounded QA—as diagnostic signals of ideological drift. Using 21,727 expert-labeled U.S. political news articles from QBias spanning left, center, and right sources, we conducted the following steps:
- Generate article-specific questions;
- Elicit document-grounded answers from three open-weight LLMs and one proprietary model;
- Detect sentence-level hallucinations via reference-based comparison;
- Classify the ideological valence of hallucinated sentences with a fine-tuned stance classifier;
- Probe output logits to relate token-level uncertainty to hallucination and drift.
Hallucination rates vary significantly across models and concentrate on contentious topics, while source-ideology differences in hallucination frequency are modest. In contrast, hallucination content exhibits robust leftward drift: a majority of hallucinated sentences are classified as left-leaning, including those generated from right-leaning sources. Logit-level analysis shows that hallucinations arise in high-entropy generation contexts, and in some models, uncertainty also predicts leftward drift, consistent with a