Modern video games combine first‑person perception, rapid visual changes, persistent world state, and heterogeneous native controls. Existing game agents map visual and task context directly to actions but lack explicit world dynamics modeling, while interactive world models only predict visual futures from supplied actions and cannot serve as task policies. World‑Action Models (WAMs) aim to unify these objectives, yet remain largely unexplored under the dynamics and open‑ended interaction of video games.
GameWAM is, to our knowledge, the first WAM for native closed‑loop gameplay and GUI control. The model jointly generates future visual observations and executable keyboard‑mouse trajectories through parallel visual and action generative processes. Core techniques include block‑causal conditioning and flow matching loss, which synchronize the temporal dependencies of vision and action streams.
To support joint learning, synchronized gameplay video sequences and GUI action trajectories are constructed. At each step the model first predicts an operation mode (gameplay or GUI), then generates actions using mode‑specific prediction distributions and applies continuous‑action normalization to accommodate heterogeneous inputs such as keyboard keys and mouse movements.
For long‑horizon interaction, GameWAM adopts block‑cycle control: it continues to predict beyond the committed horizon but only executes a short prefix of the predicted action block, replanning from new observations. Fine‑grained within‑cycle context and hierarchical cross‑cycle history preserve temporal continuity.
Experiments demonstrate competitive task success rates compared with prior agents, while executing significantly fewer native actions, indicating higher efficiency.
The authors also uncover Low‑Frequency Action Source Imprinting (LASI), where low‑frequency components of the sampled action source systematically steer coarse camera motion under fixed conditioning, revealing a source‑sensitivity failure mode in generative control.
Blogger's Review: GameWAM tightly couples world modeling with action generation, offering a more interpretable framework for game AI. The block‑causal conditioning and cross‑cycle history mechanisms are particularly innovative and merit further exploration in broader interactive settings.