Counterfactual text editing means revising a given passage in light of a hypothetical change while keeping facts that are not causally affected unchanged. For example, when asked what would happen if a doctor had not recommended surgery, the model should retain the patient’s medical history and only alter the consequences of that decision. This requires causal understanding: the model must infer, without explicit instruction, which events are impacted by the hypothetical change and which should stay fixed. Such reasoning is central to human cognition, especially in safety‑critical domains like healthcare or law. By contrast, associational text generation merely produces text that is statistically associated with the altered condition; in the medical example it might lower symptom severity because milder cases correlate with no surgery, which is plausible but not the true counterfactual of the original scenario. Existing counterfactual‑text datasets do not clearly separate these reasoning modes. To address this, we introduce the Counterfactual Benchmark (CoBe), a zero‑shot testbed for evaluating models’ ability to edit scenarios counterfactually when prompted in a conversationally natural way. CoBe comprises 2,000 questions spanning healthcare, science and engineering, business, and everyday situations. We evaluate a range of frontier and smaller models, with frontier systems achieving an average score of $58.05\%$. Findings show that models frequently default to associational generation rather than preserving the causal structure of the original scenario. Moreover, performance does not scale consistently with model size or output token length, suggesting that the failure stems more from training objectives than from insufficient model capacity.
Review