SemEval-2026 Task 5: Rating Plausibility of Word Senses in Ambiguous Stories through Narrative Understanding

AbstractWe introduce SemEval-2026 Task 5 on “Rating Plausibility of Word Senses in Ambiguous Stories through Narrative Understanding”. The dataset for this task consists of 4-5 sentence English short stories. In each story, one sentence includes a lexical ambiguity and different senses are to be judged in terms of plausibility on a Likert scale. The task is intentionally constructed to be challenging by stories only providing sparse contextual cues. We give an overview of well-performing, frequent and interesting approaches used by participating systems. From a total of 175 registered participants and 27 submitted system description papers, the best system achieved an “accuracy within standard deviation” score of 93.3%.