Language models are now widely used to convert recorded facts into media for public consumption, yet evaluation still treats source‑text agreement as the sole success metric. In reconstruction tasks, deviating from the source can be the goal; under this metric, intentional deviations are counted as errors, or, if the metric is omitted, real distortions slip through.
We tasked three model families with rewriting 2,000 news articles as either summaries or narrative stories. For each rewrite we examined changes across the five Ws (when, where, what, who, why) and the manner of expression, then judged whether each detected distortion was justified by the perspective taken. About one‑third of the distortions were deemed unjustified. Temporal and manner information was largely preserved, but location and causal links were often omitted. Accountability distortion was almost absent in summaries but rose seven‑fold in stories; among the cases where direction could be determined, two‑thirds softened the perpetrator’s responsibility. These distortions survive claim‑level fact‑checking and the retellings are often grammatically cleaner than the originals.
Judges disagreed on the justification of distortions: three authors relabeling the same instances converged on whether a distortion existed but not on whether it was warranted, highlighting a boundary issue rather than mere detection difficulty. To support research on detecting and mitigating such distortions we release the ReTold‑2k corpus, together with the scoring protocol and per‑judge baselines kept unaggregated. Repository: https://github.com/Retold2K/retold2k.
Review