PEARL (Process-Experience Aware Reinforcement Learning) improves local credit assignment for multi‑turn agents without adding a separate critic. Conventional terminal rewards give only a global signal, and group‑relative advantages provide little guidance for intermediate decisions; failed trajectories may still contain reasonable actions, and harmful decision patterns can recur during training. PEARL’s key idea is to take two continuations from the same history that end with different outcomes, locate their first divergence, and build a feature representation of the local decision at each branch. These decision features are then clustered in recent failure and success memories to form a pattern bank that captures recurring local decision motifs. Using this bank, PEARL selectively amplifies negative advantages at low‑reward decisions while leaving positive advantages and the base policy optimizer untouched. Experiments on AppWorld show improvements over ARPO on both Normal and Challenge tasks at 4B and 8B scales. On ALFWorld, PEARL reaches 97.9% mean success at 4B—2.7 points above the strongest baseline—and remains competitive at 8B. Ablation studies confirm that the shaping location, success‑bank filtering, and memory window all contribute to the effectiveness of the selective, pattern‑based updates.
Review