Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Lost in Embedding: Recovering Information Loss in Dense Retrieval

AbstractStandard dense retrievers optimize to bring relevant query document pairs closer, but do not preserve which task-critical information (e.g., entities, dates, concepts, or logical relationships) survives compression into embeddings. We observe that supplementing queries with such information at inference time improves retrieval, motivating the question of how to train information-preserving dense retrievers. We propose a new training strategy that augments contrastive learning with a reconstruction loss to provide supervision over embedding content. Our method, RecovIR, is a decoder-only LLM retriever that appends learned latent tokens and uses their representations for both cosine retrieval and autoregressive reconstruction. Across multiple architectures, RecovIR consistently improves nDCG@10 by 1–2 points on MTEB-English retrieval benchmarks over standard baselines. We find that retrieval failures often coincide with reconstruction errors, and show theoretically that when targets are relevance-sufficient and recoverable, reconstruction lower-bounds the relevance information retained in embeddings.

More from EMNLP 2026

All EMNLP 2026 papers →