Regularized self‑play is a family of methods that drive a two‑player zero‑sum policy to a Nash equilibrium by best‑responding to a slowly moving entropy‑regularized reference policy $\rho$. When the game admits a polytope of value‑equivalent equilibria, the regularizer silently breaks the tie: with a uniform reference it selects the maximum‑entropy member, i.e., the I‑projection of $\rho$ onto the Nash set. This work asks whether the reference can be deliberately used to pick a desired equilibrium.
We experiment on five analytically solvable games and a 2‑D polytope, using exact best responses and equivalence tests under fixed random seeds. By anchoring the reference at a target equilibrium and then refining, self‑play converges to that target with mean coordinate error 0.007 and median exploitability $5\times10^{-5}$, which is TOST‑equivalent to the request within $\pm0.05$. The anchoring persists through refinement and follows the reference rather than the initialization. Selection follows the reach‑weighted I‑projection with slope 0.969 (95% CI [0.950, 0.987]).
We also report failure modes: off‑manifold fixed references incur exploitability of 0.08–0.25; overly stiff or flat families require a smaller mirror step set by a pre‑registered rule; boundary targets undershoot, and curvature predicts where boundary saturation bites (rank correlation 0.90, p=0.037), while interior precision is curvature‑independent. Table and MLP steering maps are equivalent within $\pm0.03$ for every target (30 seeds). Matched control arms show that attention’s robust signature is excess seed variance, with any systematic shift bounded by 0.018 and not significant.
Against a best response, the selection‑robustness trade‑off is degenerate: steering matters only against fixed, non‑equilibrium opponents. The recipe—anchor the reference at the desired member and then refine—reinterprets the KL anchor of RLHF‑style RL as a selection knob, not merely a stability leash.
Review