Delta-Crosscoder: Robust Crosscoder Model Diffing in Narrow Fine-Tuning Regimes
Abstract
AbstractModel diffing aims to identify how fine-tuning changes a model’s internal representations. Crosscoders are attractive because they learn sparse, indexable latent dictionaries, but existing formulations struggle under narrow fine-tuning, where behaviorally important changes are sparse, low-magnitude, and dominated by shared structure. We introduce Delta-Crosscoder, which combines Dual-K BatchTopK sparsity, a delta-based loss that reconstructs activation differences through non-shared latents, and contrastive pairing of activations. Across 10 model organisms spanning synthetic false facts, emergent misalignment, subliminal learning, and taboo word guessing, Delta-Crosscoder recovers causally validated latents for all organisms, compared to 4–6 of 10 for prior crosscoder variants. The learned dictionary is also more structured: in emergent misalignment, it separates misalignment-inducing and refusal-gating directions that pooled activation-difference methods cannot disentangle. Delta-Crosscoder further maintains reconstruction quality and improves dictionary utilization in a broader base/instruction-tuned setting.