Spreadsheet applications are used by hundreds of millions worldwide, yet writing formulas remains a significant barrier. Existing approaches rely on static supervised data, which quickly saturates on limited annotations. This paper introduces FORMULASPIN, a self-play framework that breaks the ceiling of supervised fine-tuning by enabling iterative self-improvement without any additional data.
Vanilla SPIN fails on this task: it uniformly penalizes every non-matching output, leading to contradictory gradients as execution-equivalent alternatives are punished as negatives in one example while serving as ground truth in another. Our framework resolves this by leveraging the unique advantage of formula generation: binary executability provides implicit supervision that separates semantic errors from valid stylistic variants.
We frame training as a two-player game where the main player learns to prefer ground-truth formulas over those from its previous version, while execution feedback sorts outputs into distinct granularities—enabling an adaptive curriculum that shifts from semantic correctness to stylistic refinement. To further increase accuracy, we incorporate ExecVote, a semantic-level voting mechanism that naturally handles multiple valid formulations.
Experiments on multiple benchmarks demonstrate that FORMULASPIN achieves state-of-the-art performance, with a 74.9% exact match and 87.1% execution accuracy on NL2FORMULA, matching models trained with additional preference annotations while outperforming both traditional SFT and frontier proprietary models. These findings underscore self-play's potential to tackle scarce data tasks and open the door to extending it beyond executable domains.
Blogger's Review: The self-play framework of FORMULASPIN showcases an innovative approach to enhance model performance in data-scarce situations. Its unique training mechanism and the ExecVote feature are noteworthy, providing fresh insights for future NLP tasks. The success of this method may inspire more researchers to explore self-play applications in other fields.