Offline reinforcement learning with verifiable rewards (RLVR) trains on a fixed pool of sampled responses and usually supervises every token, although not all responses and tokens are equally informative. This work examines which items to supervise by analyzing the Gibbs‑optimal policy of KL‑regularized RLVR and measuring policy‑target mismatch with a bidirectional KL divergence. At the reference policy the mismatch factorizes exactly: correct responses contribute mainly through the forward KL term, focusing on their low‑probability tokens, while incorrect responses contribute via the reverse KL term, concentrating on frequently generated responses and their high‑probability tokens. Guided by this insight, we introduce GIST (Gibbs‑Targeted Hierarchical Data Selection), a two‑level selection scheme. First, the sampling temperature selects correct responses on‑policy and incorrect responses near the policy’s mode; second, the initial policy’s token probabilities are used to supervise only the tokens where the mismatch resides. Applied to two offline training objectives, GIST raises average accuracy on GSM8K, MATH‑500, and MBPP for all three models, and in a controlled Qwen3‑0.6B/GSM8K comparison it outperforms seven baselines while supervising 66.6% fewer tokens.\n\nReview: GIST’s theory‑driven hierarchical filtering cuts redundant supervision, improving data efficiency and performance of offline RLVR systems.