Unified Confidence Adjustment for Robust Cross-Modal Retrieval under Test-Time Distribution Shifts

Cross-modal retrieval models often suffer substantial performance degradation under test-time distribution shifts. Existing test-time adaptation methods primarily based on entropy minimization tend to sharpen the similarity distribution. However, retrieval relies on feature similarities to rank candidates, and the similarity gaps between nearby-ranked candidates are often small. Over-sharpening the similarity gap between the candidates can distort the semantic similarity structure, leading to miscalibrated updates and unstable retrieval. To address this issue, we propose a novel *Unified Confidence Adjustment* framework that explicitly accounts for semantic similarity within the candidate set. Specifically, we propose the *Semantic Proximity Margin* as a confidence prior. Building on this margin, we develop a *Confidence State Identification* mechanism and a *Unified Confidence Adjustment* strategy for adaptive confidence calibration. Extensive experiments on four benchmarks under both zero-shot transfer and natural corruption settings show that our proposal consistently outperforms state-of-the-art test-time adaptation methods with minimal computational overhead, underscoring the effectiveness of confidence-aware regularization for robust cross-modal retrieval.