Standard policy gradient (PG) algorithms suffer from high estimation variance because they rely on Monte Carlo updates. Off‑policy variance reduction can alleviate this issue, yet achieving optimal sample complexity usually requires restrictive assumptions such as uniformly bounded importance weights or impractical second‑order operations.\
We introduce Loopless Variance‑Reduced PG (LVR‑PG), a unified theoretical framework that abstracts a class of first‑order estimators integrating recursive updates with defensive importance sampling; known estimators like PAGE‑PG and STORM‑PG are special cases. We prove that under weak gradient domination, for any parametric policy this class attains a last‑iterate global convergence rate of $\tilde{O}(\epsilon^{-2})$. The guarantee relies solely on first‑order updates, makes minimal assumptions, and does not require gradient preconditioners.\
By extending state‑of‑the‑art lower bounds to continuous parametric policies, we obtain a sample‑complexity lower bound of $\Omega(\epsilon^{-2})$, confirming that our framework is minimax‑optimal up to logarithmic factors and horizon terms.\
Review: LVR‑PG delivers a theoretically optimal convergence rate while remaining implementation‑friendly, marking a significant step forward for off‑policy reinforcement learning theory.