Many LLM studies compare model responses under different follow‑up prompts to gauge conversational pressure. One prompt may neutrally ask the model to reconsider, while another insists that a fabricated film actually exists. An automatic scorer labels whether each response contains an unsupported factual claim, and researchers often validate the scorer with overall accuracy or $F_1$ score.
However, high overall performance can conceal prompt‑specific mistakes. If the scorer errs more after one prompt, the measured difference may be exaggerated, diminished, or even reversed. An audit dominated by difficult responses can also misrepresent performance across the full study.
We derive how such errors alter a prompt comparison and examine the issue using six models, 180 invented‑entity questions, and six prompt conditions. In a balanced audit of 288 responses, overall $F_1$ was $.916$. Yet, depending on the prompt, the scorer falsely flagged 2.6%‑33.3% of responses that the primary LLM rater judged as free of unsupported claims; these prompt‑specific estimates were uncertain.
In a second audit, the scorer and two separate LLM raters evaluated the same 300 model‑question‑prompt cases. An apparent 9.3‑point increase from asking neutrally to asserting certainty fell to 2.3 points under both raters. The primary‑rater interval included zero, so it did not establish a genuine increase. Two other prompt comparisons kept their direction under both raters, while an attempted prompt‑specific correction remained too uncertain for a firm conclusion.
Researchers should therefore validate the exact difference they plan to report, not just the scorer as a whole: audit both prompts on matching cases and include scorer and rater uncertainty in the final result.
Review