Abstract
The rapid rise of large language models (LLMs) has driven transformative adoption across enterprises. However, deploying these models in real-world settings presents unique challenges due to complex system constraints and unexpected user behaviors. Among these applications, slot filling is essential for converting unstructured input into structured, actionable data.
In this work, we introduce ESF-Bench, a challenging Enterprise Slot Filling benchmark consisting of 810 multi-turn samples and 6530 slots across 8 unique domains. Curated using a taxonomy of the 57 most challenging slot-filling scenarios observed during real-world enterprise deployments, ESF-Bench exposes notable limitations in current state-of-the-art LLMs. For instance, GPT-OSS-120b successfully extracts slots for only 20.7% of benchmark samples.
To support continued research in this area, we publicly release the benchmark dataset, taxonomy, and accompanying evaluation code on GitHub.
Blogger's Review: The release of ESF-Bench provides a crucial benchmark for slot-filling tasks in enterprise applications, revealing the limitations of current LLMs in practical applications. This will drive the development of more efficient models and improvements in real-world applications. Researchers can leverage this dataset to delve deeper into optimizing slot-filling techniques, advancing the intelligence of enterprises.