Skip to content
conftrace_
2026 ICML ICML 2026

Kalman Linear Attention: Parallel Bayesian Filtering For Efficient Language Modeling and State Tracking

State-space language models such as Mamba and gated linear attention (GLA) offer linear-complexity, parallelisable alternatives to transformers, but often lack the expressivity and robust state-tracking needed for complex reasoning. We address this by reframing sequence modeling through a probabilistic lens, with Bayesian filters as the core primitive. Classical Kalman filters provide principled state and uncertainty estimation but are typically viewed as inherently sequential. We show that reparameterising the Kalman filter in information form casts its updates as an associative scan, enabling efficient parallel training. The resulting Kalman Linear Attention (KLA) layer performs time-parallel probabilistic inference while maintaining explicit belief-state uncertainty, offering strictly more expressive nonlinear updates and gating than GLA variants while retaining their computational advantages. On language modeling tasks, KLA matches or outperforms modern SSMs and GLAs on discrete token-manipulation, state-tracking, and zero-shot commonsense-reasoning benchmarks, and is among the first stacked Bayesian-filtering primitives pretrained at the billion-token scale.

More from ICML 2026

All ICML 2026 papers →