Abstract
A small number of methodological contributions, including word2vec, the Transformer, large-scale pre-training, and reinforcement learning from human feedback, have reshaped NLP and AI research over the past decade. OpenReview now makes numeric reviewer scores and accept/reject decisions public for every ICLR submission. However, whether such review signals can identify trajectory-changing papers at submission time remains untested at corpus scale. We answer this question on $36{,}113$ papers from ICLR 2017–2025, identifying \textit{catalysts}: papers whose descendants measurably redirect future research.
We compare four disruptiveness measures (the Consolidation/Destabilization (CD) index, node2vec, the direction-aware Embedding Disruptiveness Measure (EDM), and an LLM-based semantic rater) and define a five-type operational catalyst taxonomy (topic initiator, topic bridge, within-topic redirector, simultaneous, and recognition-misaligned). EDM leads at identifying highly cited ICLR papers (AUC $0.83$ vs. $0.60$ for CD, $0.49$ for node2vec, and $0.42$ for the LLM rater). Topic initiators precede a $7.55{ imes}$ topic-share growth, and topic bridges precede an $11.52{ imes}$ growth in cross-topic citation flow versus year-matched controls. We found that the peer review scores are essentially orthogonal to future disruptiveness ($| ho|{ ext{≤}}0.005$; accepted and rejected papers have indistinguishable mean EDM, $p{=}0.11$).
Blogger's Review: This paper quantitatively reveals significant papers that influence the development of AI research, particularly the effectiveness of EDM in identifying catalysts, highlighting the limitations of traditional review scores in predicting future research impact. This finding provides crucial insights and directions for future research endeavors.