Systematic generalization—the ability to solve novel problems by recombining known atomic elements—is regarded as a core facet of human intelligence, yet it is hard to evaluate rigorously under controlled conditions. Existing benchmarks usually rely on three simplifications: (1) approximately linear action composition, which eases inductive reasoning; (2) productivity‑based tests that only probe compositional ability within the training length range; (3) action‑explicit goals that reduce abductive reasoning demands. While these shortcuts make systematic generalization easier to study, they may omit essential cognitive components.
To expose what is missed, we introduce TranSGrid, a testbed that unifies deductive, inductive, and abductive reasoning in a single task. Given a rule‑based grid with partial observations, a model must infer the hidden causal structure and generate an action sequence that satisfies a specified goal.
We evaluated seven Transformer variants on 4,800 TranSGrid instances. All models achieved up to 79.6% accuracy on a standard held‑out test set, but only 55.3% on TranSGrid, dropping to 15.8% on the hardest subset. The gap persists even within the training length range, indicating that productivity alone is insufficient to assess systematic generalization.
We then re‑introduced each of the two simplifications into TranSGrid: one makes actions compose almost linearly (reducing inductive demand), the other makes goals action‑explicit (reducing abductive demand). In both cases, solve rates recovered to roughly the level of the ordinary test set, showing that either simplification alone collapses TranSGrid into a regular held‑out benchmark.
Overall, current systematic generalization tasks tend to diminish or eliminate inductive and abductive reasoning requirements. A comprehensive measurement therefore needs a task that simultaneously engages deductive, inductive, and abductive reasoning.
Review: TranSGrid offers a more complete evaluation of systematic generalization and cautions researchers to retain all three reasoning forms when designing benchmarks.