Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Mitigating Model Merging Conflicts via Knowledge-Aware Task Vector Purification

AbstractModel merging aims to integrate task-specific abilities from individually fine-tuned models into a single model without extra training. In model merging methods, the task vector has become a fundamental building block, as it can encapsulate the information from fine-tuning. However, the merged model often suffers from performance degradation due to conflicts caused by task-irrelevant redundancy in task vectors. To address these challenges, we propose Purified TAsk VEctors (PAVE) in knowledge-aware subspace. Concretely, we sample training examples from each task, and feed them into their fine-tuned models to acquire covariance matrices before linear layers. We then perform a context-oriented singular value decomposition, which accentuates the weight components most relevant to the target knowledge. As a result, we split fine-tuned model weights into task-relevant and redundant components in the knowledge-aware subspace, and purify the task vector by pruning redundant components. To induce fair pruning efforts across models, we further introduce a spectral rank allocation strategy by optimizing a normalized activated pruning error. The task vector purification by our method as a plug-and-play scheme is applicable across various task vector-based merging methods to improve their performance. In experiments, we demonstrate the effectiveness of PAVE across diverse merging methods, tasks, and model architectures.

More from EMNLP 2026

All EMNLP 2026 papers →