Robustness of Diffusion and Autoregressive Language Models to CheckList-Style Perturbations
Abstract
AbstractWe compare diffusion-based and autoregressive language models under CheckList-style perturbations. We frame semantics-preserving perturbations as potentially containing an off-manifold component and hypothesise that diffusion denoising can reduce the resulting output change relative to autoregressive decoding. We evaluate eight instruction-tuned 7-9B models (three diffusion, five autoregressive) across sentiment, QQP, and SQuAD using behavioural pass rates and a perturbation coefficient that measures output change relative to input change. Results are task-dependent. Behaviourally, diffusion gains 18.2 and 11.0 points overall on QQP and SQuAD, while autoregressive models perform better on the sentiment and SQuAD invariance slices. On QQP, the all-model coefficient ratio does not favour diffusion ( ̂γAR/ ̂αDiff ≈ 0.91); excluding two autoregressive models with strong label-prior effects as a diagnostic sensitivity analysis yields 1.23 overall and 1.48 on invariance tests. Diffusion also performs worse on perturbations that should change the output. Thus, the observed stability differences are task-conditional, and because the empirical diffusion coefficients exceed one, the experiments do not directly validate the contraction assumption of our theoretical proposition. Data and code are available at: https://github.com/AarushRathore1729/dlmarfinal.