pretraining objective

LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference ACL 2026 Babies Learn to Look Ahead: Multi-Token Prediction in Small LMs EMNLP 2025 Pre-Training Curriculum for Multi-Token Prediction in Language Models ACL 2025 Punctuation Restoration Improves Structure Understanding without Supervision NAACL 2025 A Comparison of Language Modeling and Translation as Multilingual Pretraining Objectives EMNLP 2024 Investigating Pre-Training Objectives for Generalization in Vision-Based Reinforcement Learning ICML 2024 Tell Me What’s Next: Textual Foresight for Generic UI Representations ACL 2024 Context-Aware Transformer Pre-Training for Answer Sentence Selection ACL 2023 Downstream Datasets Make Surprisingly Good Pretraining Corpora ACL 2023 How does the task complexity of masked pretraining objectives affect downstream performance? ACL 2023 Masked Path Modeling for Vision-and-Language Navigation EMNLP 2023 Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answering ACL 2023 Procedure-Aware Pretraining for Instructional Video Understanding CVPR 2023 Team:PULSAR at ProbSum 2023:PULSAR: Pre-training with Extracted Healthcare Terms for Summarising Patients’ Problems and Data Augmentation with Black-box Large Language Models ACL 2023 UL2: Unifying Language Learning Paradigms ICLR 2023 Uncovering Hidden Consequences of Pre-training Objectives in Sequence-to-Sequence Models ACL 2023 What is the best recipe for character-level encoder-only modelling? ACL 2023 Bridging the Gap between Language Models and Cross-Lingual Sequence Labeling NAACL 2022 Effective Pretraining Objectives for Transformer-based Autoencoders EMNLP 2022 How does the pre-training objective affect what large language models learn about linguistic properties? ACL 2022 Named Entity Inclusion in Abstractive Text Summarization COLING 2022 Unsupervised Reinforcement Learning in Multiple Environments AAAI 2022 What Language Model Architecture and Pretraining Objective Works Best for Zero-Shot Generalization? ICML 2022 ConVEx: Data-Efficient and Few-Shot Slot Labeling NAACL 2021 Explicit Alignment Objectives for Multilingual Bidirectional Encoders NAACL 2021