DeoGen is a scalable automatic benchmark generator that targets deontic reasoning evaluation for language models deployed in regulated contexts. The method first auto‑formalizes policy rules into first‑order logic (using $\forall$, $\exists$) and then builds decision trees that enumerate all possible outcome trajectories. Each root‑to‑leaf path becomes a benchmark instance; leaf nodes encode a definitive verdict, while internal nodes test a model's ability to abstain when presented with insufficient information. Compared with hand‑crafted datasets, a single tree yields a combinatorial number of instances proportional to its depth, enabling massive, parametric benchmark creation. Empirical results show frontier models achieve 89%–92% accuracy on fully specified instances, but drop to 22%–40% on under‑specified cases even when explicitly instructed to abstain. In sum, DeoGen offers an automatic, scalable, and parameterizable approach to generate deontic reasoning benchmarks, dramatically reducing manual effort.
Review