This work conducts a controlled evaluation of closed‑loop LLM driving agents on the HighwayEnv (1 Hz) testbed, using gpt-4o-mini to answer when experience memory truly improves performance. The key methodological steps are:
- Treat the warm‑up experience pool as a resampled random factor, constructing ten disjoint pools;
- Add token‑matched irrelevant filler controls to neutralize prompt‑length effects;
- Audit all baseline ports, releasing seeds, caches and oracle‑certified diagnostics;
- Apply caution‑pricing metrics to quantify safety.
The probe suite spans five implementations: flat‑text retrieval (a DiLu port), a single‑frame grid port (inspired by RESPOND), an episodic temporal‑graph memory, a retrieval‑free consolidated rulebook, and a K‑step observation‑history baseline. Four main findings emerge:
- Reported memory gains stem from the warm‑up draw. Both retrieval memories beat the No‑Memory baseline only in the best of ten pools (sign test p=0.011), and the rulebook’s apparent pool‑immunity collapses on five fresh draws.
- No retrieval design surpasses the token‑matched filler baseline. A single 15‑token caution instruction reaches the collision‑only merge ceiling, and metrics that price caution erase the rulebook’s lead.
- Representational sophistication shows no robust advantage on this testbed.
- Baseline port quality can silently invert conclusions. A single‑pool comparison without filler controls would have reported memory gains that these controls reverse.
We release the testbed, audited ports, seeds, and caches for reproducibility and further research.
Review