This paper introduces an end‑to‑end approach that combines expert input with synthetic data generation to produce context‑specific benchmarks for large language models (LLMs). Existing construction methods typically trade off validity against scalability: expert‑crafted datasets offer high evaluation quality but are slow and costly, while purely synthetic data scales easily yet often yields unrealistic, redundant, or out‑of‑scope examples. To close this gap, the authors propose a schema that elicits essential information about the evaluation task’s goals, scope, and context, and uses this information to steer synthetic data creation.
Grounded in measurement validity, four dataset quality criteria are defined: coverage, diversity, content realism, and stylistic realism. These criteria demonstrate how expert‑informed scaffolds can guide synthesis toward more valid benchmarks.
Quantitative experiments compare the proposed method against purely synthetic and purely expert‑crafted baselines across several benchmarks, showing significant improvements in quality metrics while preserving validity. A real‑world case study with domain experts further validates the approach’s practicality.
The authors also analyze how different schema components affect each quality criterion and provide practical guidance on which information to prioritize when resources are limited.
Review