Replacing attention layers in a pretrained language model creates a compatibility issue: the substitute may alter representations expected by later layers. TinyCeNN-LM introduces a quality‑gated post‑training conversion framework that employs CeNN‑inspired cellular‑recurrent layers, providing bounded local processing, compact recurrent memory, routing, fusion, and an accept‑or‑rollback validation step.
The study evaluates three implementations: Integrated Memory, MemoryFusion, and PDelta3‑GDN2‑CLVR+Local32. Strict PDelta3 conversion accepts a layer only when both representation fidelity and negative log‑likelihood ($\Delta\mathrm{NLL}$) meet fixed thresholds.
On SmolLM2‑135M, layers 0‑2 are accepted with cumulative $\Delta\mathrm{NLL}=+0.01209$, while layer 3 is rejected because its representation fidelity fails despite an acceptable NLL.
On Qwen3.5‑0.8B, full‑attention layers 3, 7, and 11 pass the criteria, yielding a final $\Delta\mathrm{NLL}=+0.02073$. Integrated Memory keeps perplexity within $-0.07\%$ to $+0.93\%$ while reducing total cache by up to $6.01\%$. A sampled sanity check on 200 downstream items reports overall accuracy between $28.5\%$ and $32.0\%$ for the converted Qwen releases.
The results support a conservative, quality‑gated structural conversion approach rather than universal attention replacement or speed‑up alone.
Review