Skip to content
conftrace_
2026 EMNLP EMNLP 2026

ArabicDialectSafety: A Dialect-Aware Benchmark for Arabic Content Safety Classification

AbstractWe present ArabicDialectSafety, a human-curated Arabic safety dataset of 25,071 prompts covering six Arabic varieties—Modern Standard Arabic, Syrian, Egyptian, Algerian, Palestinian, and Moroccan—annotated with dialect labels and seven fine-grained harm categories. We introduce a dual-task evaluation framework for binary safe/unsafe detection and granular harm classification across dialects. Benchmarking seven supervised and generative models, we find that fine-tuned MARBERTv2 achieves the strongest performance (0.95 binary and 0.90 granular Macro-F1), substantially outperforming prompted frontier LLMs, including Arabic-specialized models. Our analyses show that dialect conditioning is most effective when integrated at the representation level, while significant performance gaps remain for low-resource Maghrebi dialects. We further evaluate seven frontier LLMs as response generators on harmful dialectal Arabic prompts and observe unsafe generation rates below 5% across models. We release the dataset and code upon acceptance to support future research on dialect-aware Arabic safety evaluation. Warning: this paper contains examples of harmful and potentially offensive content included solely for research purposes.

More from EMNLP 2026

All EMNLP 2026 papers →