Encoder-only and decoder-only models understand abstract argument structure differently
Abstract
AbstractLarge language models (LLMs) based on encoder and decoder architectures exhibit strong performance across a range of tasks, yet systematic comparisons of how training objectives shape their behavior remain limited. Building on prior work establishing systematic comparison between decoder-encoder differences (Weller et al., 2026), we ask whether this difference shapes different linguistic bias using two production-related tasks: semantic role assignment and syntactic structure choice (actives vs. passives). Across architectures, LLMs exhibit a robust animacy-first bias, preferring to place animate entities in subject position, diverging from human bias. However, when animacy cues are removed, encoders generalize better than decoders and show a stronger preference for passive constructions. We argue that these differences reflect training objectives: encoders’ sensitivity to contextual compatibility supports flexible mapping from semantics to structure, whereas decoders’ reliance on surface frequency favors dominant patterns and disfavors infrequent structures such as passives.