linear attention

Gated Slot Attention for Efficient Linear-Time Sequence Modeling NeurIPS 2024 In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initialization NeurIPS 2024 Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers AISTATS 2024 Linear attention is (maybe) all you need (to understand Transformer optimization) ICLR 2024 Linear Log-Normal Attention with Unbiased Concentration ICLR 2024 LongVQ: Long Sequence Modeling with Vector Quantization on Structured Memory IJCAI 2024 Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length NeurIPS 2024 MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map NeurIPS 2024 Mobile Attention: Mobile-Friendly Linear-Attention for Vision Transformers ICML 2024 Modeling the Real World with High-Density Visual Particle Dynamics CoRL 2024 Perceiving Longer Sequences With Bi-Directional Cross-Attention Transformers NeurIPS 2024 QT-ViT: Improving Linear Attention in ViT with Quadratic Taylor Expansion NeurIPS 2024 Scaling Laws for Linear Complexity Language Models EMNLP 2024 SEA: Sparse Linear Attention with Estimated Attention Mask ICLR 2024 Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences ICML 2024 Simple linear attention language models balance the recall-throughput tradeoff ICML 2024 SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization ICML 2024 Small-E: Small Language Model with Linear Attention for Efficient Speech Synthesis Interspeech 2024 The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry ICLR 2024 Understanding the Differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks NeurIPS 2024 Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attention ICML 2024 When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models ICML 2024 FLatten Transformer: Vision Transformer using Focused Linear Attention ICCV 2023 RWKV: Reinventing RNNs for the Transformer Era EMNLP 2023 Span-Selective Linear Attention Transformers for Effective and Robust Schema-Guided Dialogue State Tracking ACL 2023