Skip to content
conftrace_
2026 EMNLP EMNLP 2026

PRISM: Calibrated Prior-Risk Integration for Safe and Useful Test-Time Alignment

AbstractThis paper investigates the problem of safety alignment in large language models (LLMs) during the testing phase, particularly under jailbreaking attacks. Previous methods typically rely on identifying malicious request intentions or adjusting the LLM’s decoding strategy to inject safe behavior. However, this often leads to over-rejection, limiting their practical application. To address this, we propose an innovative framework called Prior-Risk Integration for Safe Mixture decoding (PRISM). Specifically, PRISM first uses a lightweight semantic risk evaluator to calculate the prior probability of harmfulness at the prompt level, and then continuously tracks the cumulative risk signal provided by a safety evaluator during the generation process. By comprehensively considering these two types of risks, PRISM adaptively mixes the base model’s probability distribution with a safety preference distribution, triggering a rejection response only when both the prompt prior probability and the generation risk are sufficiently high. Extensive experiments validate the effectiveness of PRISM in resisting jailbreaking attacks and its practicality. Across five jailbreaking attack methods and three neutral datasets, PRISM demonstrates superior performance compared to other baseline methods.

More from EMNLP 2026

All EMNLP 2026 papers →