Agatha: Gated Hopfield Attention for Adversarially Robust Natural Language Processing
Abstract
AbstractTransformer encoders are used for natural language understanding tasks, yet they remain vulnerable to gradient-based attacks in the embedding space. Existing defenses either use adversarial training or modify the inference pipeline, incurring additional computational overhead without a principled mechanism to contract the gradient flow. We propose Agatha, a transformer encoder that achieves adversarial robustness through architectural modifications. First, Agatha replaces softmax attention with an iterative Hopfield mechanism that projects representations into deep, sharply-bounded semantic attractors in its energy landscape. Second, Agatha introduces a gated residual connection that scales down the identity path, so adversarial gradients must propagate through the energy landscape where they contract layer by layer due to the landscape’s steep curvature.Agatha’s geometric structure contracts adversarial gradients layer by layer in the backward pass while preserving task-relevant information in the forward pass. Evaluations on standard NLP tasks demonstrate that Agatha reduces attack success rates by 13 percentage points over the best defended baseline at 4.37M scale, with the advantage growing to over 33 points at BERT-base scale (110M), without adversarial training. AutoAttack, transfer attacks, and discrete text-level evaluations confirm the robustness is not an artifact of gradient masking. Agatha adds no parameters, achieves 1.2x faster inference than the best robust baseline, and maintains clean performance within 2 points of the undefended model.