In resource‑constrained or privacy‑sensitive scenarios, in‑context learning tabular foundation models (TFMs) can solve new tasks without extra training. Yet their architecture typically stacks many transformer blocks, incurring high compute and memory costs. Traditional gradient‑boosted decision trees (GBDTs) run efficiently on CPUs, suggesting that tabular problems may not inherently require deep networks. To test whether TFMs can be simplified without losing performance, we introduce a closed‑form, label‑free affine translator that can replace a contiguous window of up to 96% of the blocks while leaving the rest untouched. Evaluating eight state‑of‑the‑art TFMs on 171 classification datasets reveals massive over‑parameterization: removing half of a model’s blocks incurs at most $0.002$ median AUROC loss across all architectures, and the half‑depth TabFM still outperforms every full‑depth TFM and tuned GBDT. Different model families over‑parameterize in distinct ways: TabPFN‑3 relies on its encoders, preserving performance regardless of which single block remains; other single‑axis models need only their first block; dual‑axis models concentrate computation in one critical block. Depth reduction can save up to 1.5 B parameters, speed up GPU inference by up to $6\times$ with a median AUROC drop of $0.03$, and a single‑block TabPFN‑3 serves cached queries $2.4\times$ faster using $6\times$ less memory at negligible cost.
Review: The work demonstrates that deep transformer stacks are unnecessary for tabular data; strategic pruning yields substantial efficiency gains while maintaining or even improving predictive performance.