Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Succinctness, Not Expressivity, Predicts Syntactic Competence

AbstractTransformers outperform state-space models and recurrent architectures on long-distance syntactic dependencies, yet no theory predicts which architecture will win on which task. For syntactic phenomena expressible as short B-RASP programs, Transformers need a fixed number of layers regardless of dependency distance, while sequential models face growing effective complexity. We derive a *succinctness prediction table* ranking Transformers, LSTMs, and Mamba on six English syntactic phenomena before any experiment, with GRU and RWKV as additional sequential controls. All predictions are confirmed: Transformer accuracy exceeds 96% at maximum structural complexity while Mamba drops below 55%, a gap that persists at 79\\times the Transformer’s parameter count. To verify the winning architecture succeeds *for the predicted reason*, Distributed Alignment Search (DAS), a causal intervention method, finds alignment of 0.95+ in Transformers but near the 0.50 chance baseline in Mamba, validated by seven families of negative controls. Training dynamics reveal that Transformers assemble the predicted algorithm in causal-dependency order, whereas Mamba achieves high accuracy without it, producing an *accuracy-alignment gap* invisible to behavioral benchmarks. Validation on Pythia confirms these patterns transfer to pre-trained models.

More from EMNLP 2026

All EMNLP 2026 papers →