Skip to content
conftrace_
2026 EMNLP EMNLP 2026

"I" can easily jailbreak LLMs

AbstractHuman-AI alignment is critical for ethical and safe use of large language models (LLMs). In this paper, we show that safety-aligned LLMs exhibit certain patterns in how refusals begin–such as disproportionately beginning responses with “I”. Can these patterns be exploited to break safety alignment? Here, we introduce ACHILLES, a lightweight RLVR-based attack that uses pattern-based verifiers to exploit these refusal-correlated behaviors, achieving substantial increases in attack success rates across diverse LLM families and exceeding 90% on safety-enhanced reasoning models, while preserving downstream capabilities. We further demonstrate that this vulnerability originates from domain-wise imbalances (safety vs non-safety) in safety alignment datasets, where safety responses share exploitable refusal-opening patterns. We additionally observe that safety-aligned models exhibit highly separable distributions between refusal and compliance responses in hidden representation space. To address this, we construct THETIS, a defense-oriented safety alignment dataset that reduces shallow refusal-pattern correlations. Models trained on THETIS maintain near-zero attack success rates against both ACHILLES and LLM-verifier based RLVR attacks, while largely preserving downstream performance.

More from EMNLP 2026

All EMNLP 2026 papers →