In this work we argue that looped Transformers degrade as the number of iterations grows and therefore require dedicated residual connections. When the loop count increases, noisy state updates overwrite correct intermediate deductions and can even undo solved sub‑problems, leaving later iterations to recover from a corrupted representation. Errors introduced early often propagate through the recurrence and become hard to correct later. To address this we introduce InfiLoop, a loop‑native residual mechanism that learns which past computations to keep and how much of each new update to accept. InfiLoop combines content‑based weighting with a learnable temporal decay to maintain a running summary of recurrent states. An exact streaming recurrence keeps the persistent aggregation memory constant regardless of loop length. The adaptive update suppresses unreliable proposals while preserving useful intermediate states. Across extensive reasoning benchmarks a 7 M‑parameter InfiLoop model outperforms existing recursive architectures, achieving 97.9% exact accuracy on Sudoku‑Extreme and 13.6% pass@2 on ARC‑AGI‑2. Notably, on Sudoku‑Extreme the model continues to improve with test‑time looping beyond 20 000 steps, demonstrating that added depth directly translates into stronger reasoning. The code is released at https://github.com/pixeli99/InfiLoop
Review