StableMask: Refining Causal Masking in Decoder-only Transformer

Qingyu Yin; Xuzheng He; Xiang Zhuang; Yu Zhao; Jianhua Yao; Xiaoyu Shen; Qiang Zhang

2024 ICML ICML 2024

StableMask: Refining Causal Masking in Decoder-only Transformer

Abstract

No abstract available. No abstract was captured for this paper when its proceedings were scraped - most often an older edition published without one online. How the corpus is built.

🌉 Interdisciplinary Bridge - Artificial Intelligence and Natural Language Processing

Authors

Qingyu Yin, Xuzheng He, Xiang Zhuang, Yu Zhao, Jianhua Yao, Xiaoyu Shen, Qiang Zhang

Topics

Machine Learning > Optimization & Theory > Neural Network Optimization Deep Learning > Architectures > Transformers Natural Language Processing > Resources & Methods > Large Language Models

Keywords

language modeling parameter-free method causal masking relative position encoding absolute positional information pseudo-attention values

Download PDF