ARJUN: A Parameter-Efficient Multi-Objective Pre-Training Framework for Language Models
Abstract
AbstractThe proliferation of large-scale language models has significantly advanced natural language processing capabilities; yet their substantial computational requirements create barriers for deployment in resource-constrained environments and limit accessibility for under-represented language communities. To address this fundamental issue, we present a novel multi-objective pre-training approach for developing parameter-efficient language models. Our method, implemented in the ARJUN model family, strategically combines four complementary pre-training objectives within a unified transformer encoder-decoder framework: (1) Masked Token Prediction (MT) for bidirectional understanding, (2) Next Token Prediction (NT) for autoregressive generation, (3) Masked Span Prediction (MS) for multi-token dependencies, and (4) Complete Sentence Prediction (CS) to bridge discriminative and generative tasks. Using this approach, our ARJUN-Base model (69M parameters) consistently outperforms standard baselines 2.5-4x larger. Extending this architecture, our scaled-up ARJUN-Large variant (up to 256M parameters) demonstrates superior architectural efficiency, establishing strong performance baselines within matched parameter budgets. Evaluations across 20 discriminative and generative tasks in Bangla, Tamil, and Hindi show prominent gains, including up to +6.5 percentage points (pp) over comparable baselines on macro-F1 scores and +17.5 pp sacreBLEU on generation. Finally, ablation studies confirm that joint training on these four objectives yields synergistic representations that consistently outperform single-task configurations across all scales.