In this study, we investigate how transformers learn modular integer multiplication, particularly the non-invertible operations over composite moduli. We propose the monoid extension: a localized generalization of Group Composition via Representation (GCR), suggesting that the learned computation does not rely on a single global representation space.
Instead, the model partitions the input space into local hierarchical algebraic regions, where group-like structure persists and Fourier mechanisms can be applied. We find that in transformers trained on square-free modular multiplication, embeddings organize around these regions, attention exhibits class-sensitive routing and low-rank write directions, and local character features explain a large fraction of the model's output logits.
Our results suggest that representation-theoretic mechanisms previously identified for group operations can extend beyond groups to more general structures.
Blogger's Review: This paper reveals the potential of transformers in handling complex mathematical operations, especially their adaptability to non-invertible operations. By introducing local algebraic structures, the study provides a new perspective for understanding the internal mechanisms of deep learning models, advancing the analysis of transformer architectures.