Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Efficient Listwise Reranking with Compressed Document Representations

AbstractReranking, the second stage of modern retrieval pipelines, is computationally expensive when performed by Large Language Models (LLMs). Existing efficiency-oriented approaches either rely on smaller models or restrict input length, both at the cost of effectiveness. Inspired by recent advances in soft document compression for retrieval-augmented generation (RAG), we introduce **RRK**, an efficient and effective listwise reranker that operates directly on multi-token compressed document representations rather than raw text. Trained via simple distillation from a strong teacher, RRK combines rich compressed representations with listwise reranking to deliver a highly efficient yet competitive system: despite its 7B-parameter backbone, it runs 5-20 faster than smaller rerankers (0.6-4B parameters) while matching or outperforming them in effectiveness. The advantage widens further on long-document benchmarks, where input compression yields the largest speedups.

More from EMNLP 2026

All EMNLP 2026 papers →