Deep research agents augment large language models with external tools to answer complex, long‑horizon questions via multi‑turn reasoning. Continuous improvement relies on learning from past experience, yet prior approaches either retrieve verbose task traces that burden decision making or distill procedural skills that remain detached from downstream policies. APEx introduces a hierarchical experience utilization framework that organizes interaction history into instance‑level trajectory memories and category‑level procedural skills, coupling them through a closed‑loop architecture of Executor, Distiller, and Planner. A three‑stage alternating GRPO training paradigm enables reward‑guided skill distillation instead of fixed‑prompt generation. At test time, distilled skills act as procedural priors for the Planner, guiding online skill‑guided reinforcement learning for ground‑truth‑free self‑improvement, with skill‑alignment regularization to prevent policy drift. Experiments on seven benchmarks show APEx achieving state‑of‑the‑art results, surpassing GPT‑5.4 by 14.7 points and the strongest memory‑augmented baseline by 3.0 points.
Review