Addressing the Concept Faithfulness Gap through Probe-Guided Sparse Autoencoders
Abstract
AbstractSparse Autoencoders (SAEs) decompose language model activations into sparse, interpretable features, but these features often struggle to predict task-relevant concepts that remain clearly encoded in the original dense activations. We call this mismatch a concept faithfulness gap: a faithful SAE should preserve concept-level representations, such as toxicity or sentiment, rather than producing features that look interpretable but lose predictive relevance. We introduce ProbeSAE, a probing-guided fine-tuning framework that addresses this gap by preserving task-relevant concept structure in a more monosemantic sparse decomposition, enabling more effective causal control over model behaviour without sacrificing interpretability. We evaluate ProbeSAE on two SAE architectures (TopK, JumpReLU), two domains (toxicity, sentiment), and multilingual out-of-distribution settings spanning six languages. Trained exclusively on English data, ProbeSAE: (i) produces more predictive, monosemantic, and interpretable sparse features than pre-trained or naïvely fine-tuned SAEs across all six evaluation languages; (ii) learns language-independent feature directions, as confirmed by cross-lingual causal ablation; (iii) passes SAEBench evaluation, confirming these gains do not arise from task/domain overfitting. Together, these results show that the concept faithfulness gap in SAEs can be addressed through probe-guided training, improving monosemanticity and causal effectiveness without sacrificing interpretability, making probe-guided SAEs a more reliable tool for domain-specific interpretability analysis.