Token-level Adaptive Weighting via Implicit Rewards for Inference-time Multi-objective Alignment
Abstract
AbstractAligning large language models (LLMs) with human preferences is challenging due to the inherent heterogeneity and multi-dimensionality of these preferences. Existing inference-time multi-objective alignment methods often rely on fixed weights to combine preference models. However, different tokens contribute unequally to preference objectives during decoding. Ignoring such token-level variation may cause the loss of preference information and lead to objective interference. To alleviate this, we utilize the implicit rewards of preference models to estimate importance differences between tokens, and propose a new training-free and reward-free framework TAW to dynamically balance multiple objectives for inference-time alignment. We design the relative entropy and marginal entropy estimators to quantify the informativeness and uncertainty of preference objectives. TAW with the estimator ensures that more informative or reliable preferences are emphasized, while weak or uncertain signals are suppressed. Experiments on two LLM alignment benchmarks show that TAW outperforms strong baselines without predefined weightings, while also achieves an overall optimal Pareto front when explicit preference weighting is applied.