SIGHT: Gaze-Guided Attention-Head Selection for Head-Sparse PEFT
Abstract
AbstractParameter-efficient fine-tuning (PEFT) reduces trainable parameters, yet head-sparse PEFT still requires deciding which attention heads should receive updates. We introduce SIGHT, **S**parse **I**dentification of **G**aze-aligned **H**eads for **T**uning, a gaze-guided head selector for head-sparse PEFT. Using eye-tracking data during reading comprehension tests, SIGHT ranks heads by correlating word-level gaze metrics with the attention heads of large language models. Specifically, the comprehension question appears either before or after the passage so that the readers either read the passage with a clear goal or for general understanding. The eye-tracking data from these two conditions are both correlated with attention heads and are fused by an offline-calibrated layer-wise gating module. When applied to Qwen3-based models, updating only the top 10% SIGHT-selected heads generally outperforms other head selectors and all-head PEFT references on reasoning and syntax benchmarks. These results suggest that gaze information during human reading can move beyond post hoc interpretation and serve as an external criterion for allocating adaptation capacity.