How Calibration Content Shapes Attention-Based Reranking
Abstract
AbstractAttention-based rerankers score documents by aggregating query-to-document attention and subtracting a null-query calibration pass to remove positional and structural bias. Although widely used, this calibration assumes that the null pass is independent of document relevance. We show that modern prompt content, e.g. constraints, instructions, personas, and demonstrations can violate this assumption when it enters the scoring readout, making the null pass relevance-aware rather than null. Through a three-pass decomposition, we find that calibration helps when the null-pass footprint is relevance-agnostic bias, but hurts when it is relevance-tracking signal: short shared constraints survive calibration, whereas long per-query instructions inject signal that subtraction destroys. We propose interpolated null calibration, a training-free modification that controls how much instruction content enters the null baseline. It recovers attention-based reranking on instruction-heavy tasks where standard calibration fails, while preserving calibration’s benefits when the null pass remains relevance-agnostic. On instruction heavy tasks, the recovered rankings surpass generative rerankers such as RankGPT. We also show that in-context demonstrations improve attention-based reranking with little calibration interference, since demonstrations act only through the query pass and leave the null pass unchanged.