Counterfactual Self-Distillation: A Distribution-Shaping Framework for Entity-Fact Unlearning in Large Language Models
Abstract
AbstractEntity-fact unlearning in large language models (LLMs) aims to remove the binding between a target entity and a forgotten fact, producing a model that behaves like one retrained without the forgotten data while preserving neighboring knowledge and overall model utility. We frame this problem as distribution shaping: such a retrained model would yield a plausible distribution at forget queries, fluent, semantically appropriate, and reflective of residual knowledge, which we take as the appropriate target. Existing objectives provide only narrow token-level or semantically uninformative supervision, falling short of this target. We propose Counterfactual Self-Distillation (CSD), a self-distillation framework that approximates the retrained distribution by constructing a counterfactual teacher through type-consistent entity substitution. The resulting teacher is decoupled from the target identity while preserving type-level relational structure, enabling directed probability reallocation without an external oracle. Experiments on TOFU and RWKU show that CSD achieves a strong forget-utility trade-off, including the best overall forgetting on RWKU and strong performance on TOFU.