This paper discusses how large language models (LLMs) synthesize game rules into executable code, referred to as Code World Models (CWMs), which are then searched by classical planners. Typically, these models are accepted when they achieve high transition accuracy on sampled trajectories. However, the authors argue that this notion of adequacy is flawed.
They present four key points:
-
An LLM-synthesized CWM can pass a sampling gate with 100% transition accuracy and have a state accuracy of over 98% on the planner's own search distribution, yet systematically fail during play.
-
This indicates that relying solely on transition accuracy does not guarantee a model's performance in real environments; the design of the model must also consider play adequacy.
-
The paper empirically demonstrates this phenomenon, highlighting the importance of focusing on both prediction accuracy and play adequacy when designing CWMs.
-
Finally, the authors propose several improvements to enhance the overall performance and reliability of CWMs.
Blogger's Review: This paper reveals that a model's performance in complex environments depends not only on theoretical accuracy but also on its real-world applicability. It points towards a multidimensional consideration in model design for future research.