Latent multi-agent systems lower inference cost by keeping intermediate reasoning and inter-agent communication in continuous representations. Conventional reusable skills are either supplied as text or stored as layer‑wise key‑value states or skill‑specific parameters, which leads to repeated computation or large per‑skill storage.
SkillMem represents each reusable skill as a small external latent memory that can be accessed directly during latent multi-agent reasoning. A role‑conditioned cross‑attention reader guides different agents with the same memory according to their roles and current hidden states.
We train the skill memories and the shared reader via behavioral distillation, matching the token‑prediction change induced by a textual skill while keeping the backbone frozen. In experiments we extract textual skills from paired successful and failed multi‑agent trajectories, though existing skills can also be compiled directly.
Across six benchmarks and Qwen3 models ranging from 4B to 14B parameters, SkillMem yields strong gains for smaller backbones while preserving efficient latent multi‑agent inference at all scales. We use a uniform $8\times256$ memory configuration; each skill, including its retrieval embedding, requires only $9.6\,KB$ of storage. For a library of 100 skills, SkillMem reduces total storage by more than $38\times$ compared with alternative representations and achieves up to a $4.23\times$ end‑to‑end inference speedup.
These results demonstrate that compact procedural memories can be reused without sacrificing the efficiency of latent multi‑agent reasoning.
Review: SkillMem offers an effective and scalable way to reuse skills via external memory, enabling large models to perform multi‑agent collaboration efficiently even under tight resource constraints.