We present a novel characterization of the reinforcement‑learning occupancy measure by embedding the planning criterion into the dynamics via a resetting planning process, whose stationary distribution we call the visitation measure. This measure is the natural object on which the information geometry of decision making is expressed. All achievable visitation measures form a dually flat statistical manifold whose two affine charts are the visitation probabilities $\pi(s,a)$ and the log‑policies $\log \pi(s,a)$, dual under the conditional entropy $H(\pi)= -\sum_{s,a}\pi(s,a)\log \pi(s,a)$.
The structure allows planning‑as‑inference to generalize from linear rewards to nonlinear functionals of the visitation measure; each iteration is solved by a single natural‑gradient step. Moreover, the temporal‑difference error (TD error) acquires the interpretation of a marginal‑utility estimate. We develop the geometry and discuss its consequences for reinforcement‑learning algorithms and theoretical neuroscience.
Review: The dually flat manifold offers a unified geometric lens for planning, preserving the duality of information theory while naturally introducing natural gradients and a TD‑error interpretation, thereby providing a solid theoretical foundation for optimizing nonlinear reward functionals.