Abstract
Reinforcement learning agents for imperfect-information card games are only as strong as their training opponents. They are hard to evaluate, as they beat random opponents over 99% of the time and only tie against copies of themselves. To address this, we built a strong, rule-based expert for Gin Rummy, using it solely as a yardstick and never for training.
This expert beats every agent we trained 70% to 99% of the time. Across more than a hundred runs, we isolate factors that make a lightweight agent stronger: trust region updates, a well-aimed reward, a curriculum of tougher opponents, warm starting, and retaining the best checkpoint. Stacking these factors lifts a self-play champion's win rate from about 30% to 36% against the expert.
However, several ideas did not pay off: short-term and long-term reward shaping, learned state embeddings, imitation learning and DAgger, and a live large language model opponent were ineffective, too slow, or too heavy to train at scale. Comparing MLP, convolutional, set-based, attention, and recurrent encoders shows that extra capacity does little to break the ceiling, suggesting the limitation lies in information rather than network size.
We add standard baselines (neural fictitious self-play and information set Monte Carlo search) and confirm that the approach carries over to Leduc Hold'em, where the optimum is computable. The result is a lightweight, game-agnostic recipe that trains competitive agents without relying on expert training, applicable to any game a small model can handle, reported with robust statistics and released as a reusable package.
Blogger's Review: This study demonstrates how to enhance the performance of lightweight game agents through clever strategies and methods without relying on complex models, emphasizing the importance of information processing over network size, providing valuable insights and directions for future game AI development.