Abstract
Large language models (LLMs) increasingly provide labels, evaluations, and feedback for tasks specified in natural language. When a specification admits multiple readings but the supervision channel does not reveal which is operative, additional labels reduce sampling error without resolving the resulting identification problem. We introduce Natural Language PAC (NL-PAC), a framework that uses a fixed model's thresholded decoding law to define admissible labels and candidate targets.
Under target-blind supervision, the probability that multiple labels are admissible equals the diameter of the pointwise-admissible target class, and every learner incurs worst-case risk of at least half this diameter at every sample size; the exact randomized minimax risk over this class is attained by a data-independent strategy. Finite-sample confidence bounds make these quantities certifiable from held-out unlabeled inputs.
In a frozen Qwen~2.5--3B audit, one prespecified prompt yields a positive model-relative certificate, whereas a paraphrase and exact-rule controls yield zero. A held-out bridge audit finds that supplied candidate reading clauses fail the admissibility condition needed to transfer the certificate to coherent readings. The guarantee is specific to the audited model, prompt, threshold, and input distribution; extending it to human interpretations requires external validation.
Blogger's Review: The NL-PAC framework offers a novel perspective on addressing specification ambiguity in LLMs, quantifying the acceptability of labels and advancing the theoretical foundation of supervised learning. Ensuring model reliability and transparency in multi-interpretation contexts emerges as a critical direction for future research.