This study investigates the performance of AlphaZero in sparsely rewarded games, particularly in two domains with contrasting structures: the solved partisan game Connect Four and the impartial game Chomp. AlphaZero has achieved superhuman performance through a neural-guided Monte Carlo Tree Search (MCTS), but strong play does not equate to perfect play. We compare vanilla AlphaZero, a multi-frame variant (limited to Chomp), and an AlphaZero Auxiliary Loss (AZAL) that incorporates oracle-derived policy supervision within a unified self-play + MCTS pipeline.
The findings indicate that vanilla AlphaZero demonstrates strong play in both domains yet fails to maintain the exact trajectories necessary for optimal play. In Connect Four, it falls short of preserving the optimal line of play, while in Chomp, it fails to consistently restore the $g=0$ invariant. Multi-frame inputs alone do not bridge this gap on rectangular Chomp boards. However, AZAL significantly improves oracle consistency across multi-seeded full-game traces and sampled-state evaluations. On Chomp, AZAL achieves perfect full-game oracle consistency on 10x11 boards and high but incomplete consistency on 9x10; in Connect Four, AZAL enhances the oracle-match rate and delays the first oracle mistake but does not reach perfect play.
Blogger's Review: This research delves into the limitations of AlphaZero in complex gaming environments and significantly enhances its strategy accuracy through auxiliary supervision mechanisms. The insights gleaned are crucial for the reinforcement learning domain and provide a roadmap for future AI gaming research.