NeFut Logo NeFut
Admin Login

[CS.AI] DreamBench-SWE: A Multi-Session Memory-Hygiene Benchmark for Software Agents

Published at: 2026-08-24 22:00 Last updated: 2026-08-29 12:04
#AI #Machine Learning

DreamBench‑SWE is a multi‑session benchmark for software‑agent memory hygiene, where later software tasks depend on non‑inferable evidence from earlier sessions and are scored by executable hidden oracles.\ \ In the original v2 fold, the primary DF‑hybrid–B5 contrast showed no significant difference (95/180 vs 89/180; clustered p=.518, Holm‑corrected p=1), which does not imply equivalence; C9/C10 remained limited by B0 headroom.\ \ The successor v2.1 audit completed 360/360 work units and 720/720 S3 cells across four conditions. Results: external memory achieved only 21/180 passes (rate 0.1167), deterministic verbatim event memory 82/180 (0.4556), typed‑plus‑raw reference probe 83/180 (0.4611), and a pinned hosted Mem0 literal‑storage configuration 97/180 (0.5389). The preregistered six‑slot Family A retained unavailable slots (p=1); all three available comparisons against no memory were rejected after Holm correction. Both preregistered mechanism contrasts were unavailable after pre‑evaluation conformance rejection. The secondary literal‑storage vs. verbatim comparison was non‑confirmatory and sensitivity‑dependent, while the reference‑probe comparison did not reject.\ \ Consequently, the audit supports DreamBench‑SWE as a discriminating executable‑profile benchmark and characterizes one exact hosted‑memory configuration, but it does not establish an external‑system mechanism, superiority among memory‑bearing conditions, equivalence, or broad product generality. The original v2.0.5 findings and artifacts remain unchanged.\ \ Blogger's Review: This benchmark offers a nuanced evaluation framework for multi‑session memory dependencies in software agents. Although the current study does not demonstrate a clear advantage for external memory solutions, it highlights critical comparison points and methodological rigor for future investigations.

Original Source: https://arxiv.org/abs/2608.20664

[h] Back to Home