Lingo-Blocksworld: A Dataset for Linguistically-Informed Blocksworld
Abstract
AbstractWe introduce Lingo-Blocksworld, a linguistically-informed diagnostic benchmark for evaluating the generalisation and planning abilities of Large Language Models (LLMs) in the Blocksworld (BW) domain. Unlike prior obfuscated BW variants, our method replaces content words with morphosyntactically well-formed nonsense words, preserving the grammatical scaffolding of the domain while removing lexical familiarity. This enables us to test a psycholinguistically-motivated hypothesis, and find systematic support across both reasoning and non-reasoning models, with degradation patterns that mirror syntactic bootstrapping from human language processing. Additionally, we test the ability of models to generalise planning solutions from the typical 4-block problems to 12 blocks. Overall, our results demonstrate that reasoning models stay robust where non-reasoning models collapse: non-reasoning model performance drops to near zero once problems exceed four blocks or have actions and predicates obfuscated, while reasoning models maintain strong performance until structural and lexical difficulty are combined. Manual error analysis pinpoints further differences. Reasoning model errors are more often skipped steps or incomplete instruction following, whereas non-reasoning models fail to derive plausible plans at all. By combining psycholinguistic motivation with classical planning evaluation, Lingo-BW allows us to shift research focus from whether LLMs plan to how their planning depends on linguistic surface form.