The recent EyeBench benchmark reveals a stark gap in predicting reading comprehension from eye movements: text-aware models using pretrained language models achieve 56–63% AUROC, while gaze-only models operate at chance level.
We investigate how far a gaze-only model can be enhanced through lightweight, language-model-free conditioning. Building on the EyeBench AhnCNN baseline, LEXIC-Base, we propose two mechanisms to inject three precomputed word-level difficulty signals—GPT-2 surprisal, word frequency, and word length—into the per-fixation input.
The two mechanisms are direct concatenation (LEXIC-Concat) and a residual mechanism (LEXIC-Res), where a small head predicts typical-reader gaze response and the encoder is conditioned on the deviation.
In the OneStop reading comprehension task, with K=5 seed-ensemble training across ten folds, both mechanisms yield statistically significant AUROC gains on Unseen Text, increasing by 1.8 to 2.2 percentage points, with Wilcoxon p values indicating significance.
Blogger's Review: LEXIC significantly enhances gaze-based models' performance in reading comprehension tasks by introducing word-level difficulty signals. This innovation not only showcases the potential of lightweight models but also provides new avenues for future research, particularly in effectively leveraging gaze data without language models. The statistical significance of the results underscores the method's efficacy, making it worthy of exploration and application in other domains.