ProtLingo is an efficient protein language model framework that augments a pretrained single‑sequence Transformer backbone with conditional local memory and sparse expert routing. The model first maps contextual residue representations into route‑specific discrete codes, then composes centered local windows into latent $N$‑gram addresses and retrieves reusable residual signals associated with recurring local sequence contexts. In parallel, selected feed‑forward blocks are upcycled into sparse Mixture‑of‑Experts layers, sharing experts that are routed per residue so that only a subset of parameters is activated, enabling residue‑dependent computation. Experiments on protein fitness prediction, the FLIP benchmarks, and supervised contact prediction show that ProtLingo achieves competitive performance with a 150 M‑scale backbone, offering strong parameter efficiency on mutation‑effect prediction while preserving long‑range structural representations.
Review