softmax attention

Cumulant Attention in Vision Transformers (Student Abstract) AAAI 2026 Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selection JMLR 2026 ReHyAt: Recurrent Hybrid Attention for Video Diffusion Transformers CVPR 2026 Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling ACL 2026 LETS Forecast: Learning Embedology for Time Series Forecasting ICML 2025 Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers ICML 2025 Superiority of Multi-Head Attention: A Theoretical Study in Shallow Transformers in In-Context Linear Regression AISTATS 2025 Transformers are Provably Optimal In-context Estimators for Wireless Communications AISTATS 2025 When Attention Sink Emerges in Language Models: An Empirical View ICLR 2025 A Theoretical Understanding of Self-Correction through In-context Alignment NeurIPS 2024 Bridging the Divide: Reconsidering Softmax and Linear Attention NeurIPS 2024 In-context Convergence of Transformers ICML 2024 In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness NeurIPS 2024 MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map NeurIPS 2024 One-Layer Transformer Provably Learns One-Nearest Neighbor In Context NeurIPS 2024 The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry ICLR 2024 Understanding the Differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks NeurIPS 2024 Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization NeurIPS 2024 BiViT: Extremely Compressed Binary Vision Transformers ICCV 2023 Efficient Attention via Control Variates ICLR 2023 cosFormer: Rethinking Softmax In Attention ICLR 2022 Linear Complexity Randomized Self-attention Mechanism ICML 2022 Rethinking Attention with Performers ICLR 2021