Retrieval‑Augmented Generation (RAG) improves large language models by grounding their responses in external knowledge, but multi‑hop reasoning is prone to error propagation: a failure in an early retrieval step contaminates later reasoning and often leads to a wrong final answer. Conventional outcome‑based optimization only rewards the final answer, leaving intermediate retrieval and reasoning mistakes invisible. Existing process‑based methods add step‑level signals but still score each step against the final answer, which can mistakenly reward flawed steps that happen to produce the correct answer.
PRO‑STEP addresses this limitation by evaluating both logical validity and evidential grounding at every step. Its main components are:
- Generative Process‑Rating Model (PRM) – takes the current step text and its retrieved evidence as input and outputs two scores: one for logical soundness, another for evidence match.
- PRM‑guided value‑tree search – constructs preference pairs in the search space, pairing a valid step with a flawed one, and uses the PRM’s dual scores to distinguish them.
- Step‑level Direct Preference Optimization (DPO) – updates the generation policy directly from the preference pairs, encouraging the model to select steps that the PRM deems superior.
Experiments on single‑ and multi‑hop QA datasets covering five benchmarks show that PRO‑STEP achieves the highest average Exact Match (EM) and F1 scores, outperforming both pure outcome‑reward baselines and earlier process‑reward approaches. The code, models, and training data are publicly released at https://github.com/keemminnke/PRO-Step.
Review: By integrating dual‑dimensional step evaluation with preference‑based direct optimization, PRO‑STEP effectively mitigates error accumulation in RAG pipelines, offering a practical route to more reliable knowledge‑grounded generation.