MoH: Multi-Head Attention as Mixture-of-Head Attention

Peng Jin; Bo Zhu; Li Yuan; Shuicheng Yan

2025 ICML ICML 2025

MoH: Multi-Head Attention as Mixture-of-Head Attention

Abstract

No abstract available. No abstract was captured for this paper when its proceedings were scraped - most often an older edition published without one online. How the corpus is built.

Authors

Peng Jin, Bo Zhu, Li Yuan, Shuicheng Yan

Topics

Artificial Intelligence > Core AI > Foundation Models Machine Learning > Application Areas > Efficient Computing Deep Learning > Architectures > Transformers

Keywords

multi-head attention vits llama3-8b expert heads mixture-of-head attention weighted summation

Download PDF