TRACER: Trace-Regularized Training Against Indirect Prompt Injection
Abstract
AbstractIndirect prompt injection exploits a core tension in LLM systems: models must use retrieved documents, tool outputs, and agent messages that are useful but untrusted. Existing defenses are often trained and evaluated within a single presentation wrapper, implicitly assuming that robustness transfers across framings. We show that this assumption is unsafe: for a model trained to resist malicious follow-up messages, the same injected instruction succeeds only 18% of the time as a follow-up message, but 53% of the time when embedded in retrieved context. Thus, robustness is not payload-only; it is channel-conditioned. We address this with TRACER (Trace-Regularized Adversarial Channel Exposure with Reward-shaping), a channel-aware adversarial training algorithm that treats the wrapper as part of the optimization problem. TRACER combines channel-sampled adversarial exposure with two training components: an answer-anchored, evidence-gated composite reward for post-attack behavior, and ATRC, a trace-regularization loss that aligns attacked reasoning with clean anchor spans when injected evidence is invalid. On GSM8K with Qwen2.5-3B-Instruct, TRACER reduces RAG-framed ASR from 65.5% to 6.1%, while preserving 78–82% clean accuracy. Cross-framing experiments reveal a depth–breadth trade-off: RAG-specialized training gives the strongest native defense, tool-output training transfers most uniformly, and joint RAG+tool training achieves the best 3B average ASR of 13.9%. At 7B scale, a single epoch reaches 9.4% average ASR. TRACER substantially narrows the framing gap, highlighting the need to train and evaluate defenses across deployment channels.