NeFut Logo NeFut
中 Admin Login

[CS.AI] ImplicitHop: Evaluating Knowledge Integration Across Context and Memory

Published at: 2026-11-03 22:00 Last updated: 2026-11-05 00:45
#AI #LLM #Knowledge Integration

Integrating newly acquired information and deducing its consequences is essential for learning, understanding, and reasoning in large language models. Whether the information arrives during pre‑training or is supplied in‑context, a model must store local facts and propagate their long‑range effects globally. To study this capability we introduce two complementary natural‑language benchmarks for large‑scale evaluation of deductive reasoning as a function of inference depth. The first benchmark is a formal graph suite that allows controlled testing across diverse graph topologies and integration tasks. The second benchmark composes knowledge from real‑world knowledge graphs, reflecting practical relational structures. Using these benchmarks we compare knowledge integration from context versus from model weights on a set of open‑source models, all without chain‑of‑thought prompting. Results show that accuracy drops sharply as composition length grows for both pretrained and in‑context knowledge, yet in‑context information consistently outperforms deduction over weight‑encoded facts. These findings quantify a fundamental limitation of current models, with direct implications for prompt‑based behavior control, continual‑learning strategies, and the design of training, prompting, and integration techniques that support deeper reasoning.\n\nReview: The paper delivers a rigorous, quantitative framework that highlights the superiority of in‑context cues for deep reasoning, offering valuable guidance for future work on enhancing the synergy between a model’s long‑term memory and its immediate inference abilities.

Original Source: https://arxiv.org/abs/2610.41169

[h] Back to Home