Junk DNA Hypothesis: Pruning Small Pre-Trained Weights $\textitIrreversibly$ and $\textitMonotonically$ Impairs “Difficult" Downstream Tasks in LLMs

Lu Yin; Ajay Kumar Jaiswal; Shiwei Liu; Souvik Kundu; Zhangyang Wang

2024 ICML ICML 2024

Junk DNA Hypothesis: Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs “Difficult" Downstream Tasks in LLMs

Abstract

No abstract available. No abstract was captured for this paper when its proceedings were scraped - most often an older edition published without one online. How the corpus is built.

🌉 Interdisciplinary Bridge - Artificial Intelligence and Natural Language Processing

Authors

Lu Yin, Ajay Kumar Jaiswal, Shiwei Liu, Souvik Kundu, Zhangyang Wang

Topics

Machine Learning > Core Methods > Representation Learning Natural Language Processing > Resources & Methods > Large Language Models Artificial Intelligence > Core AI > Large Language Models

Keywords

downstream task continual training pretrained weight pruning magnitude difficulty spectrum

Download PDF