SALM: Social Arabic Language Model
Abstract
AbstractArabic NLP faces a fundamental mismatch between training and deployment conditions. While most language models are trained predominantly on Modern Standard Arabic (MSA), real-world communication occurs largely in spoken dialects. To bridge this gap, we leverage social media content from Reddit, Twitter, and YouTube as a scalable proxy for spoken Arabic. We construct a 1.04-billion-token pretraining corpus and train SALM, a compact 336M-parameter decoder-only language model pre-trained via next-token prediction alone, no masked language modelling, no dialect labels, no task-specific signal. Despite an order-of-magnitude smaller training corpus than existing dialectal PLMs and a generative training objective that provides no direct classification signal, SALM matches MARBERT, explicitly pre-trained on one billion dialectal tweets, within 0.7 percentage points on MADAR-26 dialect identification accuracy and outperforms it on macro-F1. Under full fine-tuning, SALM surpasses all baselines, achieving 69.1% accuracy on 26-way city-level dialect classification. Under the same frozen probe on binary Arabic sentiment (AJGT), SALM achieves 92.2% accuracy, topping all Arabic PLMs, encoder and decoder alike. We release the corpus and trained model to facilitate future research on Arabic spoken language modelling.