kv cache

Eigen Attention: Attention in Low-Rank Space for KV Cache Compression EMNLP 2024 InfiniPot: Infinite Context Processing on Memory-Constrained LLMs EMNLP 2024 IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact ACL 2024 KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization NeurIPS 2024 NACL: A General and Effective KV Cache Eviction Framework for LLM at Inference Time ACL 2024 PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference ACL 2024 QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference ICML 2024 SCA: Selective Compression Attention for Efficiently Extending the Context Window of Large Language Models EMNLP 2024 SGLang: Efficient Execution of Structured Language Model Programs NeurIPS 2024 SnapKV: LLM Knows What You are Looking for Before Generation NeurIPS 2024 Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression ACL 2024 S^3: Increasing GPU Utilization during Generative Inference for Higher Throughput NeurIPS 2023 H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models NeurIPS 2023 Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time NeurIPS 2023