Large language models excel at code generation for mainstream programming languages but struggle with rare, domain-specific languages such as MiniZinc, a constraint modeling language for combinatorial problems. We investigate whether targeted fine-tuning can teach small language models (0.6B to 20B parameters) to generate syntactically correct and semantically valid MiniZinc models from natural language problem descriptions.
Our key finding is that syntax errors dominate failures when working with this domain-specific language: the out-of-the-box execution accuracy of small language models such as Qwen3, LLaMa, Gemma, and GPT-OSS is near-zero.
We propose a cross-model error bootstrapping approach that collects syntax errors from multiple LLM runs and leverages those to curate an error correction training dataset. This dataset allows us to fine-tune small language models that consistently improves both direct code generation and chain-of-thought approaches across all model sizes.
With self-reflection and ensembling, our approach achieves up to 98% execution accuracy. In parallel, solution accuracy still remains at 35%, indicating that while syntax is learnable, constraint reasoning remains a challenge. We contribute our fine-tuning pipeline, datasets, and models to open-source for further research on text-to-model translation.
Blogger's Review: This study showcases the potential of small language models in specific domain tasks, particularly their effectiveness in syntax correction. While constraint reasoning remains a challenge, the open-source fine-tuning pipeline and datasets pave the way for future advancements in this area. Looking forward to seeing more research outcomes on tackling constraint reasoning challenges!