inference optimization

Eigen Attention: Attention in Low-Rank Space for KV Cache Compression EMNLP 2024 Fast and Memory-Efficient Video Diffusion Using Streamlined Inference NeurIPS 2024 Faster Vocoder: a multi threading approach to achieve low latency during TTS Inference Interspeech 2024 Fast Matrix Multiplications for Lookup Table-Quantized LLMs EMNLP 2024 FlattenQuant: Breaking through the Inference Compute-bound for Large Language Models with Per-tensor Quantization COLING 2024 Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Use ACL 2024 In-Context Former: Lightning-fast Compressing Context for Large Language Model EMNLP 2024 Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following AAAI 2024 KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization NeurIPS 2024 KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization NeurIPS 2024 LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding ACL 2024 LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback NAACL 2024 LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference EMNLP 2024 LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models ICLR 2024 MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention NeurIPS 2024 MOSEL: Inference Serving Using Dynamic Modality Selection EMNLP 2024 NACL: A General and Effective KV Cache Eviction Framework for LLM at Inference Time ACL 2024 Normalized Narrow Jump To Conclusions: Normalized Narrow Shortcuts for Parameter Efficient Early Exit Transformer Prediction EMNLP 2024 Optimized Speculative Sampling for GPU Hardware Accelerators EMNLP 2024 Overload: Latency Attacks on Object Detection for Edge Devices CVPR 2024 Predicting Rewards Alongside Tokens: Non-disruptive Parameter Insertion for Efficient Inference Intervention in Large Language Model EMNLP 2024 PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference ACL 2024 Query-OPT: Optimizing Inference of Large Language Models via Multi-Query Instructions in Meeting Summarization EMNLP 2024 QUIK: Towards End-to-end 4-Bit Inference on Generative Large Language Models EMNLP 2024 Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs ACL 2024