Lost in Embedding: Recovering Information Loss in Dense Retrieval
Abstract
AbstractStandard dense retrievers optimize to bring relevant query document pairs closer, but do not preserve which task-critical information (e.g., entities, dates, concepts, or logical relationships) survives compression into embeddings. We observe that supplementing queries with such information at inference time improves retrieval, motivating the question of how to train information-preserving dense retrievers. We propose a new training strategy that augments contrastive learning with a reconstruction loss to provide supervision over embedding content. Our method, RecovIR, is a decoder-only LLM retriever that appends learned latent tokens and uses their representations for both cosine retrieval and autoregressive reconstruction. Across multiple architectures, RecovIR consistently improves nDCG@10 by 1–2 points on MTEB-English retrieval benchmarks over standard baselines. We find that retrieval failures often coincide with reconstruction errors, and show theoretically that when targets are relevance-sufficient and recoverable, reconstruction lower-bounds the relevance information retained in embeddings.