Abstract
Constraining the generation of autoregressive large language models (LLMs) is an important component of integrating language models into formal systems. In the generation of code and data for tasks like program synthesis, ensuring that language models produce syntactically valid output is a prerequisite for processing such output. These languages (such as SQL or JSON) are often designed as $LR(k)$ context-free grammars.
By distilling the LLM to a tractable probabilistic model, its autoregressive generation can be steered and masked to incorporate the probability of satisfying logical constraints, ensuring high quality output that is guaranteed to be valid. This paper demonstrates that the satisfaction of any $LR(k)$ grammar of finite duration can be calculated in polynomial time, an improvement over the exponential time of applying previous methods to such grammars. This result enables efficient constraint and steering of LLM generation towards output that better satisfies formal syntactic constraints.
Blogger's Review: This paper presents an efficient method for constraining the generation of autoregressive language models, significantly improving the efficiency of generating syntactically valid outputs. By transforming complex context-free grammar problems into solvable forms in polynomial time, it establishes a foundation for code generation and data processing in practical applications, providing substantial theoretical and practical value.