Evolution Strategies (ES) enable memory‑efficient full‑parameter fine‑tuning of large language models (LLMs) using only forward passes. Standard ES, however, averages rewards uniformly across tasks and collapses population feedback into a single scalar, which makes it hard to capture how the learning value of each task changes as the model improves. To overcome this, we introduce DART‑ES (Difficulty‑Aware Reweighting and Targeted Replay for Evolution Strategies).
DART‑ES estimates the local solvability of each task by measuring the pass rate of the perturbation population on that task, and aggregates historical observations to build a shared dynamic difficulty state. This state jointly drives (1) continuous difficulty reweighting, encouraging the model to focus on tasks whose difficulty drops most rapidly, and (2) targeted replay of rare solvable samples, improving utilization of the few successful instances. The whole mechanism requires no extra difficulty predictor and no back‑propagation.
Extensive experiments show that DART‑ES outperforms vanilla ES on all five base models, raising average accuracy from 72.07% to 73.53% and surpassing GRPO’s 73.26% on GSM8K. On five challenging mathematical‑reasoning benchmarks, DART‑ES achieves a 49.20% average accuracy, modestly above ES’s 48.34% and competitive with strong 7B models trained with reinforcement learning. Additional tests on instruction tuning, code generation, and the Countdown task with a 14B model demonstrate strong cross‑task generalization and scalability to larger models.
Beyond accuracy gains, DART‑ES offers clear system‑efficiency benefits: compared to GRPO, per‑step runtime drops by 15.2%–50.2% and peak GPU memory usage decreases by 21.1%–51.1%. Even though it performs full‑parameter fine‑tuning, DART‑ES requires less runtime and memory than the GRPO + LoRA combination.
Review: DART‑ES achieves a rare combination of performance and efficiency improvements by leveraging difficulty‑aware dynamic reweighting and targeted replay, highlighting the promise of evolution strategies for large‑scale model fine‑tuning.