Formal contracts are essential for software testing and verification, yet writing them remains labor-intensive and error-prone. Large Language Models (LLMs) offer a promising path toward autoformalization: synthesizing executable assertions from natural-language specifications, thereby bridging the gap between informal developer intent and formal executable specifications. We present Monty: an autoformalization framework for assertions that tackles the challenges of expectations of validity of assertions and ambiguity in natural language. Our techniques are based on filtering formalizations using a novel conformance score metric and validity scores obtained from testing the code against formalized assertions. We evaluate our approach on 541 assertion-generation tasks derived from 22 collection-like Java classes and show that our technique produces the ground truth more reliably (improving up to 20 points in precision on average) than when using LLMs naively to translate assertions.
Blogger's Review: The introduction of the Monty framework presents a fresh perspective in the software testing domain, especially as automation continues to enhance efficiency and reduce human errors. The innovative approach based on conformance scoring is worthy of further exploration and application, showcasing the immense potential of LLMs in practical development.