This paper investigates how efficiently Transformers can store factual knowledge and what implications this has for model design and system limits. Previous work reported a linear relationship between the number of memorized facts and the number of model parameters, far worse than what theory suggests is achievable. To re‑examine empirical scaling, we conduct systematic experiments with extensive Bayesian hyper‑parameter optimization, identifying the smallest model that can perfectly memorize a given fact set and measuring how this minimum size scales with the number of facts.
On synthetic datasets composed of random, independent (subject, relation, object) triples—e.g., (Paris, capital_of, France)—with no exploitable structure, we find that memorization capacity—the largest dataset a model can fully memorize—grows exponentially with the Transformer dimension. Equivalently, the minimum dimension $d_{\min}$ required to memorize $N$ facts grows logarithmically: $d_{\min}\propto \log N$. Across 11 dataset sizes ranging from $N=45$ to $49{,}920$, a lack‑of‑fit F‑test does not reject the logarithmic fit (with an offset), while all other alternatives, including power‑law, are rejected.
Along the depth axis, keeping width fixed, the maximal memorizable dataset size increases sublinearly with the number of layers over the measured range. We further extend the study to structured data containing symmetric relations or two‑hop compositions. In this setting, Transformers exploit these regularities to further boost capacity. All code to reproduce the results is available at https://github.com/Magronox/transformer_information_capacity.
Review: The work provides a rigorous experimental characterization of Transformer memory limits, revealing a logarithmic scaling law that deepens our theoretical understanding and suggests that increasing width may be more effective than simply deepening networks for capacity gains.