In the emerging task of context learning, LLMs (Large Language Models) must learn and apply novel, task-specific knowledge from intricate contexts absent during pre-training; even frontier models show less than 24% task success. This work conducts a comprehensive empirical study to understand why this setting remains challenging. A natural hypothesis is that failures stem from content access; however, across twelve retrieval, reflection, and verification baselines on CL-Bench, an extensive context learning benchmark, we find limited gains over direct full-context prompting.
Further failure analysis reveals a key finding: unlike typical long-context tasks such as long document understanding, context learning requires not only recovering local content but also acquiring local specifications that are often unspecified in the query but distributed across the context: domain-specific formats, local rules, and completeness conditions. Among all 31,592 rubric items, we find that 55.4% clearly evaluate specification acquisition, while only 22.6% evaluate content acquisition.
Moreover, despite 76.7% of specifications being unspecified in the user query, 95.5% are traceable to the context, indicating these are learnable obligations rather than hidden requirements. To validate this diagnosis, we design a deliberately simple intervention called PSCI (Private Specification-Contract Induction) which extracts local specifications and enforces them through adversarial checking and repair; PSCI achieves a state-of-the-art 28.14% (+5.59 pp absolute and +24.8% relative) on CL-Bench, replicated on Qwen3.5-27B (+5.28 pp) and Gemini 3 Pro (+6.17 pp). Seventeen ablations further isolate the role of task-specific specifications.
Overall, our results suggest that context learning hinges on not only content acquisition but also specification acquisition.
Blogger's Review: This paper provides an in-depth empirical exploration of the significance of specification acquisition in context learning, with the proposed PSCI method offering new insights for future context learning tasks. Although the current success rates remain low, this research points the way toward understanding and improving the reasoning capabilities of LLMs. Specifically, effectively extracting and utilizing specification information from contexts will be key to enhancing model performance.