Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models

Samira Abnar; Harshay Shah; Dan Busbridge; Alaaeldin El-Nouby; Joshua M. Susskind; Vimal Thilak

2025 ICML ICML 2025

Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models

Abstract

No abstract available. No abstract was captured for this paper when its proceedings were scraped - most often an older edition published without one online. How the corpus is built.

🌉 Interdisciplinary Bridge - Artificial Intelligence and Natural Language Processing

Authors

Samira Abnar, Harshay Shah, Dan Busbridge, Alaaeldin El-Nouby, Joshua M. Susskind, Vimal Thilak

Topics

Artificial Intelligence > Core AI > Foundation Models Machine Learning > Application Areas > Efficient Computing Natural Language Processing > Resources & Methods > Large Language Models

Keywords

sparsity scaling law model performance mixture-of-experts flops parameter

Download PDF