Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining

Qi Zhang; Tianqi Du; Haotian Huang; Yifei Wang; Yisen Wang

2024 ICML ICML 2024

Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining

Abstract

No abstract available. No abstract was captured for this paper when its proceedings were scraped - most often an older edition published without one online. How the corpus is built.

❓ The Questioner

Authors

Qi Zhang, Tianqi Du, Haotian Huang, Yifei Wang, Yisen Wang

Topics

Machine Learning > Learning Types > Self-Supervised Learning Deep Learning > Models > Generative Models

Keywords

classification task autoregressive ssl content generation tasks fixed position tokens flexible tokens masked ssl

Download PDF