2022
ACL
ACL 2022
Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model
Abstract
AbstractEnsembling is a popular method used to improve performance as a last resort. However, ensembling multiple models finetuned from a single pretrained model has been not very effective; this could be due to the lack of diversity among ensemble members. This paper proposes Multi-Ticket Ensemble, which finetunes different subnetworks of a single pretrained model and ensembles them. We empirically demonstrated that winning-ticket subnetworks produced more diverse predictions than dense networks and their ensemble outperformed the standard ensemble in some tasks when accurate lottery tickets are found on the tasks.
Authors
Topics
Artificial Intelligence > Core AI > Model Compression
Deep Learning > Techniques > Model Architecture
Deep Learning > Techniques > Pretraining
Machine Learning > Learning Types > Transfer Learning
Machine Learning > Core Methods > Ensemble Methods
Deep Learning > Optimization & Theory > Model Compression
Deep Learning > Learning Types > Ensemble Learning