Large language models (LLMs) are increasingly acting as research agents, yet evaluating their ability to track shifts in research attention is challenging because reviews and ideas lack uniquely verifiable outcomes. To address this, we introduce Research Attention Prediction (RAP), a rolling benchmark that spans 278 AI/ML fields and 1,390 episodes. At each cut‑off, an LLM agent searches a temporally restricted arXiv corpus and predicts the paper share for the next six months across eight frozen research directions.
Search generally helps, but all four diagnostic models perform worse than an exact‑count exponential weighted moving average (EWMA) baseline in compositional accuracy. We identify two linked bottlenecks: under cumulative‑history access, State carry‑forward outperforms direct Forecast for all models; frozen‑evidence replay ties this reversal to Forecast‑oriented policies that retrieve a smaller share of recent evidence. Even with perfect historical activity, future‑specific updating remains limited—only GPT‑5.5 combined with reopened Search slightly surpasses EWMA.
Fine‑tuning on realized outcomes yields notable gains. After fine‑tuning, Qwen3‑4B’s forecast Spearman correlation improves by 0.105 on held‑out fields at later origins, with additional benefits on change‑rich episodes.
Review: RAP offers a systematic platform for assessing LLMs’ ability to track research attention, exposing subtle trade‑offs between retrieval strategies and historical utilization, and pointing to concrete avenues for model improvement.