Skip to content
conftrace_
2026 EMNLP EMNLP 2026

CHUNDER: Enabling Million-Token Qwen3 Serving on AMD ROCm

AbstractMillion-token context serving is becoming a critical requirement for industry LLM systems for real deployments. However, practical 1M-context serving for open-source models has remained heavily CUDA-centric, largely because the required long-context runtime features are not fully available on AMD ROCm. With AMD GPUs now reported at around 19% market share, this gap matters. We address it by enabling 1M-token Qwen3 serving on AMD GPUs with ROCm. We present , a runtime stack that enables million-token serving for the widely used open-source Qwen3 family on AMD ROCm. achieves this by intercepting vLLM’s attention routing at runtime and redirecting queries, keys, and values to JIT-compiled Triton Dual Chunk Attention (DCA) kernels, without forking the serving engine. We then harden the surrounding runtime to make full 1M-token execution practical: DCA-aware paged metadata, tiled single-request chunked prefill, MoE token chunking that reduces per-rank transient memory from approximately 91.6 GiB to 10.6 GiB, and FP8 GEMMs with hardened hipBLASLt kernel selection. We evaluate across multiple million-token context tasks and show consistent serving performance and exact retrieval correctness. To our knowledge, this is the first reported end-to-end Qwen3 million-token serving path on AMD ROCm integrated with vLLM.

More from EMNLP 2026

All EMNLP 2026 papers →