This study examines how prompt framing influences both factual statements and tone in large language model (LLM) statistical analysis reports. We used a 4×4 factorial design with four framing conditions – neutral request, critical framing (search for disconfirming evidence), significance‑seeking framing (search for confirming evidence), and a simplified mix of the first two – crossed with four ground‑truth data patterns: a genuine effect, a confound that mimics an effect but fails a robustness check, a well‑powered null, and an under‑powered null. A total of 480 model responses were collected and scored on two independent dimensions: whether the factual claim diverged from the correct interpretation, and whether only the tone diverged while the claim remained correct.
The factual errors clustered in two scenarios. When a genuine effect was paired with critical framing, the model adopted unwarranted skepticism (97% error). When an under‑powered null was paired with significance‑seeking framing, the model overstated confidence in a null conclusion the data could not support (100% error). Tone shifts were far more pervasive: critical framing produced a defensive, hedge‑heavy register across all data patterns, whereas significance‑seeking framing altered tone only where the data were genuinely ambiguous. The presence of a confound in the data virtually eliminated both kinds of shift under every framing condition.
These results indicate that framing‑induced distortion in LLM‑assisted data analysis is not uniform; it depends on both the prompt tone and the underlying data characteristics. A model can retain a correct conclusion while its tone varies substantially around it.
Review