Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging

Pierre Ablin; Angelos Katharopoulos; Skyler Seto; David Grangier

2025 ICML ICML 2025

Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging

Abstract

No abstract available. No abstract was captured for this paper when its proceedings were scraped - most often an older edition published without one online. How the corpus is built.

🌉 Interdisciplinary Bridge - Artificial Intelligence and Natural Language Processing

Authors

Pierre Ablin, Angelos Katharopoulos, Skyler Seto, David Grangier

Topics

Artificial Intelligence > Core AI > Foundation Models Machine Learning > Application Areas > Domain Adaptation Natural Language Processing > Resources & Methods > Large Language Models

Keywords

language modeling pretraining linear combination domain weight specialist model expert parameters

Download PDF