NeFut Logo NeFut
中 Admin Login

[CS.AI] Can Language Models Play the Wikipedia Game?

Published at: 2026-10-13 22:00 Last updated: 2026-10-14 02:30
#AI #Machine Learning #LLM

Language models are increasingly employed to retrieve information from document collections that are too large to index, a task that often demands specialized linguistic or scientific knowledge and is therefore challenging.\ \ We used the Wikipedia Game as a benchmark to test models on a comparable information‑retrieval problem. In the game, players must reach a target article by following only the hyperlinks within articles.\ \ We evaluated sentence‑embedding models and large language models (LLMs) across five Wikipedia languages and five academic‑literature datasets. To avoid reliance on third‑party APIs, we built massive local graphs of Wikipedia pages and paper citations using OpenAlex and released the code for constructing and navigating these graphs.\ \ Model performance was compared against human players and a TF‑IDF baseline. No model significantly outperformed humans, but the top systems—GPT‑o3 mini, GPT‑4o, QwQ:32B, and DeepSeek‑R1:70B—achieved success rates comparable to human levels.\ \ Performance varied across languages and topics, offering practical insights for practitioners choosing retrieval techniques.\ \ Review: The Wikipedia Game serves as a unified, reproducible evaluation platform that clearly reveals the current strengths and weaknesses of language models in multilingual, cross‑domain navigation tasks, guiding future improvements.

Original Source: https://arxiv.org/abs/2610.13948

[h] Back to Home