Is Alignment Enough for Helpfulness, Harmlessness, and Honesty?
Abstract
AbstractAligning Large Language Models (LLMs) with the objectives of helpfulness, harmlessness, and honesty (HHH) remains important for reliable deployment. Existing approaches commonly rely on Supervised Fine-Tuning (SFT) and Mixture-of-Experts (MoE) for alignment. Prior work has identified Axis Collapse as a failure mode where different alignment objectives occupy disjoint feature spaces, with SFT showing interference between conflicting objectives, i.e. catastrophic forgetting, while MoE models show miscalibrated routing due to unreliable expert selection. Reinforcement Learning from Human Feedback (RLHF) has been applied to mitigate this, though it frequently shows reward misattribution, where alignment signals disproportionately favor certain objectives over others. To address these issues, we propose TriAlignX, a two-stage framework for HHH alignment. In Stage 1, low-rank task vectors are extracted for each alignment objective and routed through PrefSelect, a query-conditioned policy that selects relevant objectives for each query. In Stage 2, an encoding module maps the selected task vectors into objective-specific alignment scores, while a response module produces alignment-aware embeddings for decoding. Experimental results show that TriAlignX achieves up to 97.85% helpfulness (↑), 26.65% harmlessness (↓), and 98.90% honesty (↑), with an Avg score of 55.65%. On DeepSeek-7B, TriAlignX further achieves 54.96% Avg while reducing inference latency to 0.68s and peak memory usage to 13.2GB. Our code is available at: https://github.com/gskgautam/TriAlignX