The Continuous Memory Machine (CMM) is a recurrent architecture that employs matrix‑valued short‑term and long‑term memory states for distinct functional roles. Built on the Continuous Thought Machine (CTM), the short‑term memory continuously tracks recent neural activity, while each neuron is equipped with a uniquely parameterized model that leverages these activity patterns for computation. A persistent long‑term memory stores information for later retrieval. A Transformer jointly updates both stores, providing a bidirectional read‑write mechanism: each store can reorganize its own contents and also read from and write to the other. Across algorithmic, in‑context learning, and recurrent reasoning benchmarks, CMM outperforms a wide range of baselines, showing stronger generalization than previous memory‑augmented networks while preserving the interpretable attention patterns of CTM. The implementation is available at https://github.com/SakanaAI/continuous-memory-machines.
Review