transformer optimization

AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurations COLING 2025 Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformers EMNLP 2025 LAWCAT: Efficient Distillation from Quadratic to Linear Attention with Convolution across Tokens for Long Context Modeling EMNLP 2025 Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization EMNLP 2025 p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decay ICCV 2025 The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergence ACL 2025 TreeKV: Smooth Key-Value Cache Compression with Tree Structures IJCAI 2025 CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling EMNLP 2024 FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision NeurIPS 2024 KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches EMNLP 2024 Linear attention is (maybe) all you need (to understand Transformer optimization) ICLR 2024 MemoryFormer : Minimize Transformer Computation by Removing Fully-Connected Layers NeurIPS 2024 Context Compression for Auto-regressive Transformers with Sentinel Tokens EMNLP 2023 Leap-of-Thought: Accelerating Transformers via Dynamic Token Routing EMNLP 2023 Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention EMNLP 2023 Sequence Parallelism: Long Sequence Training from System Perspective ACL 2023 You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Model CVPR 2023 Speeding up Transformer Decoding via an Attention Refinement Network COLING 2022 Faster Depth-Adaptive Transformers AAAI 2021 Optimizing Deeper Transformers on Small Datasets ACL 2021 Value-aware Approximate Attention EMNLP 2021 Improving Transformer Optimization Through Better Initialization ICML 2020 Making Asynchronous Stochastic Gradient Descent Work for Transformers EMNLP 2019 Marian: Cost-effective High-Quality Neural Machine Translation in C++ ACL 2018