MAGMA-Bench is an architecture‑agnostic benchmark designed to assess persistent task reasoning for long‑horizon embodied agents operating in dynamic multi‑robot settings. The suite progressively stretches task horizons and injects controlled online perturbations—execution failures, task interruptions, and mission updates—while requiring coordination among heterogeneous robots with distinct skill sets. A structured skill‑level interface separates high‑level reasoning from perception and low‑level control, focusing evaluation on decision making.
We benchmark locally deployable open‑weight planners, multi‑agent state‑tracking methods, and state‑of‑the‑art frontier models. Even the strongest models solve only about two‑thirds of clean episodes, and their success rates drop further under online disturbances. Supervised adaptation markedly improves a small model, yet performance remains substantially lower on longer and multi‑robot missions.
The benchmark releases 261,646 task‑level training examples and an architecture‑agnostic expert‑trace generation pipeline for reproducibility and future research.
Review