Skip to content
conftrace_
2026 EMNLP EMNLP 2026

HyDRA: Hybrid Dynamic Routing Architecture for Heterogeneous LLM Pools

AbstractProduction LLM deployments maintain heterogeneous model pools spanning order-of-magnitude cost differences, yet existing routers make binary strong-vs-weak decisions coupled to specific model identities, requiring retraining on every catalog change. We present HyDRA (Hybrid Dynamic Routing Architecture), which uses a lightweight ModernBERT encoder to predict capability requirements (reasoning, code generation, debugging, and tool use) for each query. It then selects the cheapest model meeting those requirements through shortfall matching. HyDRA decouples the learned predictor from the model catalog, enabling catalog changes without retraining. On SWE-Bench Verified, HyDRA is 0.3 QR points below always-strong (not statistically significant) at 54.1% cost savings. On a held-out multilingual evaluation set, the same checkpoint retains 80.6% of oracle-routing quality at 37.5% cost savings, tunable to 79.1% retention at 56.1% savings. A 14-day A/B flight with close to 1M users per arm found statistically significant improvements in latency, time-to-first-token, reliability, and user activity. Deployed across GitHub Copilot VS Code Chat, GitHub Copilot CLI, and GitHub Copilot extensions, HyDRA demonstrates cross-lingual routing consistency across 16 languages and 4 language groups.

🌉 Interdisciplinary Bridge - Artificial Intelligence and Natural Language Processing

More from EMNLP 2026

All EMNLP 2026 papers →