Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Token-level Adaptive Weighting via Implicit Rewards for Inference-time Multi-objective Alignment

AbstractAligning large language models (LLMs) with human preferences is challenging due to the inherent heterogeneity and multi-dimensionality of these preferences. Existing inference-time multi-objective alignment methods often rely on fixed weights to combine preference models. However, different tokens contribute unequally to preference objectives during decoding. Ignoring such token-level variation may cause the loss of preference information and lead to objective interference. To alleviate this, we utilize the implicit rewards of preference models to estimate importance differences between tokens, and propose a new training-free and reward-free framework TAW to dynamically balance multiple objectives for inference-time alignment. We design the relative entropy and marginal entropy estimators to quantify the informativeness and uncertainty of preference objectives. TAW with the estimator ensures that more informative or reliable preferences are emphasized, while weak or uncertain signals are suppressed. Experiments on two LLM alignment benchmarks show that TAW outperforms strong baselines without predefined weightings, while also achieves an overall optimal Pareto front when explicit preference weighting is applied.

More from EMNLP 2026

All EMNLP 2026 papers →