Bifurcated Attention for Single-Context Large-Batch Sampling

Ben Athiwaratkun; Sujan Kumar Gonugondla; Sanjay Krishna Gouda; Haifeng Qian; Hantian Ding; Qing Sun; Jun Wang; Jiacheng Guo; Liangfu Chen; Parminder Bhatia; Ramesh Nallapati; Sudipta Sengupta; Bing Xiang

2024 ICML ICML 2024

Bifurcated Attention for Single-Context Large-Batch Sampling

Abstract

No abstract available. No abstract was captured for this paper when its proceedings were scraped - most often an older edition published without one online. How the corpus is built.

🌉 Interdisciplinary Bridge - Artificial Intelligence and Natural Language Processing

🐣 Hot Topic Early Bird - kv cache

Authors

Ben Athiwaratkun, Sujan Kumar Gonugondla, Sanjay Krishna Gouda, Haifeng Qian, Hantian Ding, Qing Sun, Jun Wang, Jiacheng Guo, Liangfu Chen, Parminder Bhatia, Ramesh Nallapati, Sudipta Sengupta, Bing Xiang

Topics

Machine Learning > Application Areas > Efficient Computing Deep Learning > Architectures > Transformers Natural Language Processing > Resources & Methods > Large Language Models

Keywords

kv cache latency bifurcated attention gemm operations memory io costs single-context batch sampling

Download PDF