Skip to content
conftrace_
2026 EMNLP EMNLP 2026

DarterSD: Dynamic Token-Weighted Distillation for High-Efficient Speculative Decoding

AbstractSpeculative decoding accelerates autoregressive LLM generation with a lightweight draft model, yet existing distillation methods treat all historical tokens equally despite their divergent contributions to next-token prediction. We discover that attention in draft models is extremely concentrated: the top-3 historical tokens capture 80% of attention weight, while over 90% of tokens contribute negligibly. This uniform training paradigm forces the draft model to learn from noisy, low-importance tokens, diluting the learning signal from informative positions and limiting the alignment between draft and target distributions. To address this, we propose DarterSD, a dynamic token-weighted distillation framework for efficient speculative decoding. DarterSD employs a learnable sigmoid gate in self-attention to compute per-token importance scores, and designs a multi-objective distillation loss comprising distribution alignment, logit matching, and gate regularization to focus training on high-informative tokens. We further introduce a lightweight Boolean-mask denoising strategy at inference to suppress contextual noise, without modifying the target model’s verification pipeline. Experiments show DarterSD delivers a lossless speedup up to 5.52x , outperforming EAGLE3 (4.67x), and boosts the average accepted token length by 12.9%.

More from EMNLP 2026

All EMNLP 2026 papers →