Skip to content
conftrace_
2026 EMNLP EMNLP 2026

REIGN: Refurbished Embeddings with Integrated Guidance Networks for Efficient Context-Length Scaling

AbstractDense retrieval over long documents is expensive. Token-level encoders scale quadratically in sequence length, and most long-context embedding models reach 32K tokens only through architectural workarounds or by stretching billion-parameter LLMs. We propose REIGN (Refurbished Embeddings with Integrated Guidance Networks), a contrastively trained bi-encoder that operates on sequences of contextualised chunk embeddings from a frozen Guidance Network (GN) rather than on raw tokens. REIGN targets multi-chunk inputs and primarily document-to-document retrieval; single-chunk inputs stay with the GN. Decoupling token-level processing from document-level reasoning, and caching the GN embeddings to disk, cuts per-document training cost by roughly four orders of magnitude relative to chunked Transformer fine-tuning. We also release a synthetic long-document retrieval benchmark for contrastive training and evaluation at long context lengths. Across an in-distribution Wikipedia benchmark, the LoCo out-of-distribution suite, and a real-world patent retrieval case study, REIGN matches dense long-context retrievers at smaller parameter budgets per regime. A paired significance test puts it on par with models 1.6–4.3× larger on the patent task, and it stays within 0.65 nDCG@10 of a 20×-larger model on LoCo.

🌉 Interdisciplinary Bridge - Artificial Intelligence and Computer Science and Natural Language Processing

More from EMNLP 2026

All EMNLP 2026 papers →