Skip to content
conftrace_
2026 EMNLP EMNLP 2026

A3: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Serving

AbstractLarge language models (LLMs) have demonstrated strong capabilities in processing long contexts, enabling them to tackle tasks involving long textual inputs such as multi-turn conversations, legal documents, or retrieved documents in Retrieval-Augmented Generation (RAG) systems. However, despite their ability to handle long sequences, the resulting decoding latency and memory overhead remain substantial, posing challenges for real-world deployment. Recent advances in KV Cache reuse have shown potential to mitigate these costs, but these methods still suffer from notable performance degradation. To address this issue, we conduct an in-depth investigation of recomputation-based reuse methods and observe that the recomputed tokens often fail to align with the context segments most relevant to the question. This misalignment hinders proper updates to the critical contextual representations. Therefore, we propose the Attention-Aware Accurate KV Cache Fusion algorithm (A3), which precomputes and selectively fuses the KV Cache of text chunks based on their relevance to the question, achieving accurate integration with minimal computational overhead. Extensive experiments on various benchmarks and LLMs demonstrate that A3 achieves competitive task performance compared to five baselines while reducing the time-to-first-token (TTFT) by 2×.

More from EMNLP 2026

All EMNLP 2026 papers →