LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens

Yiran Ding; Li Lyna Zhang; Chengruidong Zhang; Yuanyuan Xu; Ning Shang; Jiahang Xu; Fan Yang; Mao Yang

2024 ICML ICML 2024

LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens

Abstract

No abstract available. No abstract was captured for this paper when its proceedings were scraped - most often an older edition published without one online. How the corpus is built.

🌉 Interdisciplinary Bridge - Artificial Intelligence and Natural Language Processing

Authors

Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, Mao Yang

Topics

Natural Language Processing > Resources & Methods > Large Language Models Artificial Intelligence > Core AI > Large Language Models

Keywords

embedding context window fine tuning llms positional interpolation progressive extension

Download PDF