Skip to content
conftrace_
2026 EMNLP EMNLP 2026

SproutRAG: Attention-Guided Tree Search with Progressive Embeddings for Long-Document RAG

AbstractRetrieval-augmented generation (RAG) systems must balance retrieval granularity with contextual coherence, a challenge that existing methods address through LLM-guided chunking, single-level context expansion, or hierarchical summarization. These approaches variously depend on costly LLM calls during indexing or retrieval, limit context aggregation to a single granularity level, or introduce information loss through summarization. We present SproutRAG, an attention-guided hierarchical RAG framework that builds a binary tree over sentence-level chunks using learned inter-sentence attention from a sentence-level large language model (SLLM). To avoid proximity bias from uniform attention averaging, SproutRAG learns head- and layer-level aggregation weights that identify attention patterns most aligned with semantic document structure. At retrieval time, it uses hierarchical beam search over the tree to gather candidates at multiple granularities, capturing multi-sentence relevance beyond flat retrieval. The framework is trained end-to-end with a joint objective that improves both embeddings and tree structure, without requiring external LLM calls. Experiments across four benchmarks spanning scientific, legal, and open-domain settings demonstrate that SproutRAG improves information efficiency (IE) by 6.1% on average over the strongest baseline.

More from EMNLP 2026

All EMNLP 2026 papers →