Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapters

Yuhang Zhou; Zihua Zhao; Siyuan Du; Haolin Li; Jiangchao Yao; Ya Zhang; Yanfeng Wang

2024 ICML ICML 2024

Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapters

Abstract

No abstract available. No abstract was captured for this paper when its proceedings were scraped - most often an older edition published without one online. How the corpus is built.

Authors

Yuhang Zhou, Zihua Zhao, Siyuan Du, Haolin Li, Jiangchao Yao, Ya Zhang, Yanfeng Wang

Topics

Machine Learning > Application Areas > Domain Adaptation Deep Learning > Architectures > Transformers Artificial Intelligence > Core AI > Large Language Models

Keywords

heterogeneous data low-rank adapter large language model mixture of low-rank adapters (mola) task identifiers task-wise decorrelation loss

Download PDF