Training long‑horizon language model agents requires assigning credit to intermediate decisions from sparse terminal feedback. Conventional approaches often rely on a global critic or extra reward models, yet useful comparisons between decisions in existing rollouts depend on their context, the evidence for each alternative, and how that evidence is fed into learning.
We introduce Matched‑Context Group Policy Optimization (MCGPO), a procedure that builds weighted action preferences from already collected trajectories. MCGPO first defines comparison contexts using pre‑action state summaries and observed rubric progress, annotated by task rules. A two‑action support mechanism selects between exact matching and coarse matching rules, establishing ordered comparisons within a finite group of sibling rollouts.
To avoid leakage, MCGPO separates candidate decisions from the terminal outcomes used to orient their preferences into fixed folds. Within each fold, evidence‑dependent weights are computed; these weights scale an auxiliary pairwise objective alongside the shared GraphGPO task learning loss.
The main advantage is that MCGPO reuses rollout data and the full history of each response, preserving a critic‑free training pipeline. Experiments span six environment‑backbone settings, each with three random seeds. Results show:
- On ALFWorld and WebShop, with Qwen2.5‑1.5B and 7B models, MCGPO improves mean success over matched GraphGPO by 0.47‑1.46 percentage points.
- With larger Qwen3‑4B and 8B models on WebShop, gains reach 3.96 and 5.16 points respectively.
- ALFWorld’s baseline success already exceeds 92%, leaving limited headroom yet still achieving noticeable improvement.
- Integrated success rises in three of four Qwen2.5 settings.
Three‑seed factorial controls confirm the benefit of opposite‑fold evidence and evidence weighting; across all four representation‑control settings, joint state‑rubric matching attains the highest mean performance.
Review