Skip to content
conftrace_
2026 EMNLP EMNLP 2026

NARRATOR: Narrative-Aware Routing and Evaluation of Large Language Models

AbstractNarrative NLP encompasses a diverse set of competencies such as detecting whether a text contains a narrative, classifying its genre or function, completing its plot, and judging similarity between stories. However, the relationship between model capabilities across these sub-tasks remains poorly understood and no unified benchmark exists to evaluate large language models across these dimensions simultaneously. In this paper, we introduce NARRATOR, a unified benchmark of 96,818 multiple-choice questions derived from six narrative datasets spanning detection, classification, story completion, and similarity. Evaluating 50 open-weight LLMs in a zero-shot setting reveals that LLM performance profiles are highly heterogeneous and no model dominates across all sub-tasks. To exploit this complementarity efficiently, we learn compact narrative competence embeddings of LLMs using a matrix factorisation over a binary correctness matrix, without accessing model weights. A single linear layer on top of these frozen embeddings supports model routing (71.1%), correctness forecasting (65.3%), and benchmark accuracy prediction (mean Kendall’s 𝜏 = 0.72) at near-zero inference cost.

🌉 Interdisciplinary Bridge - Artificial Intelligence and Natural Language Processing

More from EMNLP 2026

All EMNLP 2026 papers →