NeFut Logo NeFut
中 Admin Login

[CS.AI] Benchmarking LLM Next-Step Recommendations with Knowledge Space Theory

Published at: 2026-11-12 22:00 Last updated: 2026-11-13 06:56
#AI #Machine Learning #LLM

This paper introduces a benchmarking approach based on Knowledge Space Theory (KST) to evaluate large language models (LLMs) for next‑step recommendations in educational tasks. The procedure works as follows: given a task catalogue and an existing prerequisite relation, KST is used to generate all admissible knowledge states and the complete set of immediate next steps for each state. Under a specified prompt, the model’s recommendation set is produced and compared element‑wise with these reference sets.\ In a sample study, 2,400 responses were collected from six models across four mathematics domains, four prompt conditions, and two catalogue orders. Exact‑set accuracy reached 61.3% when only descriptions were provided, 62.8% when models were asked to reconstruct prerequisites, 96.3% with supplied prerequisite relations, and 98.2% with the full knowledge space. Performance varied across models and domains, and recommendation sets agreed in 85.25% of paired presentations.\ The benchmark enables context‑specific comparison of model‑prompt combinations in terms of structural recommendation accuracy, but the scores do not imply learning benefit nor prove that an LLM is necessary for task selection.\ Review: This framework offers a quantifiable and reproducible way to assess LLMs’ reasoning over knowledge structures, aiding researchers in identifying genuine capabilities beyond surface‑level performance.

Original Source: https://arxiv.org/abs/2611.13667

[h] Back to Home