This paper investigates the distinction between learning a fact and learning how to retrieve it. Models behave differently under various request forms; for example, after the prompt "The capital of X is Y" they can produce "Y", but they may fail when the prompt is "The capital of X:". To separate fact learning from retrieval, the authors propose a two‑stage training protocol. In stage one (request‑form training), model A is exposed to each fact in five different request forms, while model B sees the same facts only as plain statements. In stage two (target‑fact training), both models are trained on new facts presented solely as statements. Results show that both models retrieve the new facts equally well from statements, yet they diverge sharply on other request forms. An analysis of the hidden representation just before answering—the context state—reveals that the model trained on multiple request forms produces more similar context states across different prompts than the statement‑only model. Manipulating this state at retrieval time can enable or block the recall of an already learned fact, and the effect transfers across relations such as capitals and currencies. Moreover, intervening on the context state only during target‑fact training alters later retrieval performance, demonstrating that the context state during fact acquisition plays a crucial role. In summary, a model’s retrieval ability depends jointly on early experience with request forms and the context state during fact learning.
Review