Abstract
Reinforcement learning for deep-search agents has largely focused on trajectory-level scoring, including outcome correctness, citation-aware rewards, and evidence coverage. However, the actions that expose supporting documents receive no targeted credit, a gap we call the reward-credit mismatch. We propose STAMP, in which a reference-based verifier judges whether each cited document supports an entity or relation in a training-time evidence graph.
Method
In STAMP, first-exposure attribution traces each supported citation back to the action that first surfaced it. This step credit is injected through sign-preserving advantage modulation, redistributing advantage across steps without changing the trajectory-level reward or the relative ranking of trajectories within each group.
Experimental Results
On BrowseComp, BrowseComp-ZH, and xbench-DS, STAMP improves the GRPO baseline by +2.0/+5.5/+3.0 points under matched SFT initialization, training data, and search tools, and composes with both outcome-only and citation-rubric base rewards. Component ablations confirm that the provenance-based credit signal and the sign-preserving advantage modulation each contribute to the gains.
Blogger's Review: The STAMP method addresses the reward-credit mismatch issue, offering a fresh perspective for deep search agents, especially in terms of citing supporting documents. Its innovative approach to provenance-based credit assignment enhances the precision and efficiency of the learning process. I hope to see more research like this advancing the field further.