EYELIGNR: Aligning Gaze and Text via Contrastive Learning for Improved Reading Scanpath Prediction
Abstract
AbstractModeling the interaction between gaze dynamics and linguistic structure remains a central challenge in scanpath prediction. Existing approaches either incorporate linguistic information in ways that do not fully respect the temporal progression of reading, or ignore linguistic structure altogether. Their effectiveness is further limited by the scarcity and heterogeneity of available eye-tracking data. To address these issues, we introduce EYELIGNR, a framework for spatiotemporal scanpath prediction that integrates linguistic information at the representation level. EYELIGNR conditions language representations on fixation order and aligns gaze and text within a shared embedding space through contrastive learning. Experiments on five reading corpora show that EYELIGNR generalizes effectively across reading conditions, improving over prior state of the art on multiple scanpath metrics (e.g., up to NLD ~57%, NLL ~27%, and FLA ~59%).