When Helpfulness, Harmlessness, and Honesty Remember They Are on the Same Team
Abstract
AbstractAligning Large Language Models (LLMs) requires simultaneously satisfying Helpfulness, Harmlessness, and Honesty (HHH). Existing works-including SFT, MoEs, RLHF, and orchestrator-based steering-operate on standard 1-to-1 transformer architectures and show systematic trade-offs across objectives. While such works can steer behaviour post hoc, they remain fundamentally additive and therefore suffer from catastrophic forgetting. While recent mechanistic interpretability have helped characterise this behaviour, they expose a deeper structural failure-transformers lack an mechanism to negotiate incompatible objective signals, resulting in a mechanistic blind-spot. Therefore, we test the hypothesis that these failures are architectural-driven rather than optimisation-driven via NegotiaX, a 1-to-N transformer architecture that separates and reconciles alignment objectives via two modules. Module I (Transformer Module) allocates independent representational pathways to each objective-mitigating catastrophic forgetting. Module II (Mechanistic Alignment Circuit (MAC)), embeds objectives into dedicated subspaces and performs a learned negotiative operation prior to decoding-mitigating mechanistic blind-spot. Across four 7B backbones, NegotiaX remains competitive with SOTA-improving Avg by up to 6.3%, Helpfulness by 6.2%, Honesty by 5.6%, and Harmlessness by 4.7%.