Skip to content
conftrace_
2026 EMNLP EMNLP 2026

A Benchmark for Arabic Lemma-Level Readability Assessment

AbstractWe present BAREC-LexBench, a benchmark for Arabic lemma-level readability assessment, designed to evaluate readability lexicons and computational models independently of their granularity levels. We build the benchmark through a large-scale pairwise comparison task, where annotators determine which of two Arabic lemmas is easier to read and understand. The benchmark consists of 36,000 annotated pairs covering both same- and cross-part-of-speech comparisons among open-class lemmas: adjectives, nouns, proper nouns, and verbs. The annotations were produced by expert Arabic educators with an average inter-annotator agreement of 77.7% (0.56 Kappa ∼ moderate agreement), and we analyze disagreement patterns to better understand where lemma-level readability judgments are most challenging. We benchmark a wide range of approaches, from readability lexicons and frequency-based lexicons to trainable models and large language models (LLMs). Finally, we demonstrate downstream uses of the benchmark by constructing a ranked readability lexicon covering open-class lemmas (BAREC-LexRank) and evaluating its contribution to sentence-level readability assessment. To support research and education, we make the dataset, ranked lexicon, and benchmark results openly available.

More from EMNLP 2026

All EMNLP 2026 papers →