Think You Have Solved Commonsense Reasoning? Try HellaSwagUltra
Abstract
AbstractAs LLMs advance, popular commonsense and NLU benchmarks have become saturated, yet multilingual evaluation still lags. Existing benchmarks cover only a small set of languages, and translation based multilingual benchmarks can introduce bias. We introduce HellaSwagUltra, a multilingual commonsense and NLU benchmark with substantial local cultural knowledge. It is built with an automated, continuously extensible pipeline. Instead of explicitly prompting for reasoning, each example implicitly embeds two commonsense or local knowledge facts in the context, and answer options contain subtle cues about whether those facts are violated, requiring models to pick the most plausible continuation. All test items are fully reviewed and corrected by language experts, with ongoing updates. Even the strongest proprietary model (e.g., Gemini-2.5-Pro) reaches only 62.5% accuracy, while GPT-4o and leading open-source models hover around 40%–50%, demonstrating that multilingual commonsense reasoning remains an open challenge.