Skip to content
conftrace_
2026 EMNLP EMNLP 2026

HSEARCH-R1: Learning Compact Value Models for LLM-guided Hierarchical Search via RL

AbstractReasoning-intensive retrieval often requires evidence whose relevance is not captured by keyword or embedding similarity alone, exposing a core limitation of traditional IR systems. An emerging direction known as LLM-guided hierarchical search addresses this limitation by letting an LLM directly traverse a semantic tree over the corpus. Such methods so far work well only with a frontier LLM in the search loop, which makes inference expensive. We propose HSearch-R1, a distillation recipe for learning compact (4B) LLMs that perform efficient LLM-guided hierarchical search. The recipe distils a frontier teacher’s per-step value estimates from its search trajectories into a much smaller LLM (Qwen3.5-4B) via RL, using a composite reward that matches the teacher’s ranking and score distribution at each step; this outcome-level RL distillation substantially outperforms supervised fine-tuning on teacher outputs. We further propose a retrieval-prior-informed variant of the search that injects a cheap retriever’s shortlist as a prior into the tree and adds robustness when the offline index is stale. The base HSearch-R1-4B matches or closely tracks frontier LLMs on the seven static BRIGHT-StackExchange subsets at roughly 50× lower per-query cost. Adding the retrieval prior brings HSearch-R1 to 52.5 nDCG@10 on StackExchange and 45.6 across all 12 BRIGHT subsets, within 1 nDCG of the strongest published ensembled baselines while remaining substantially cheaper.

🌉 Interdisciplinary Bridge - Artificial Intelligence and Reinforcement Learning

More from EMNLP 2026

All EMNLP 2026 papers →