Transformers keep an explicit history of the sequence, allowing attention to retrieve past representations directly. However, the KV‑Cache size and the cost of processing a new token grow linearly with context length. State Space Models compress the past into a fixed‑size recurrent state, at the expense of reduced addressability.\n\nWe introduce the Dynamical Memory Transformer (DMT), a hybrid architecture that separates sequence encoding from information reading. The encoder consists of several memory units, each realized as a dynamical system with a complex diagonal recurrent matrix. An input‑dependent leak rate controls their updates. The decoder then uses attention to read and combine the units’ outputs without traversing the token history.\n\nThe recurrence formulation admits an associative scan with total work $O(T)$ and parallel depth $O(\log T)$, while keeping dimensions fixed. During autoregressive inference, both the recurrent states and the convolution cache remain constant in size, yielding constant‑cost attention.\n\nWe compare DMT with Mamba and the standard Transformer on cognitive benchmarks and language‑modeling tasks. The results show that DMT achieves the best scores on several cognitive tasks, attains language‑modeling performance close to the Transformer but still trails Mamba, and exhibits stronger length extrapolation on the selective‑copy task than both baselines.\n\nThese findings demonstrate that parallel dynamical memory units constitute a promising mechanism for retaining and exploiting information over long sequences.\n\nReview: DMT decouples recurrent memory from attention to achieve fixed‑cost processing of long contexts, excelling in reasoning‑oriented tasks while leaving room for improvement in large‑scale language modeling.