key-value cache

InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management OSDI 2024 KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache ICML 2024 KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation ICML 2024 Layer-Condensed KV Cache for Efficient Inference of Large Language Models ACL 2024 LoCoCo: Dropping In Convolutions for Long Context Compression ICML 2024 Loki: Low-rank Keys for Efficient Sparse Attention NeurIPS 2024 LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference EMNLP 2024 MiniCache: KV Cache Compression in Depth Dimension for Large Language Models NeurIPS 2024 Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantization EMNLP 2024 Reducing Transformer Key-Value Cache Size with Cross-Layer Attention NeurIPS 2024 RelayAttention for Efficient Large Language Model Serving with Long System Prompts ACL 2024 SnapKV: LLM Knows What You are Looking for Before Generation NeurIPS 2024 You Only Cache Once: Decoder-Decoder Architectures for Language Models NeurIPS 2024 Context Compression for Auto-regressive Transformers with Sentinel Tokens EMNLP 2023 Focused Transformer: Contrastive Training for Context Scaling NeurIPS 2023 Long-Range Language Modeling with Selective Cache EMNLP 2023 Segcache: a memory-efficient and scalable in-memory key-value cache for small objects NSDI 2021 Flashield: a Hybrid Key-value Cache that Controls Flash Write Amplification NSDI 2019 A Simple Cache Model for Image Recognition NeurIPS 2018 LHD: Improving Cache Hit Rate by Maximizing Hit Density NSDI 2018