Skip to content
conftrace_
2026 EMNLP EMNLP 2026

LLM Mental Health Response Manipulation

AbstractLarge language models (LLMs) are increasingly used in conversations involving emotional distress, where responses should remain empathetic, validating, exploratory and safe. We investigate whether system prompts and user-level framing can manipulate these qualities in mental-health and interpersonal-harm scenarios. Our focus is on whether small open-weight models express empathy consistently, without assuming that an empathetic response is necessarily safe or appropriate all the time. Using 50 risk-stratified posts sampled from Reddit support communities we evaluate Llama 3.1 8B, Qwen3 4B and Gemma 3 4B across five system-prompt conditions and a complementary role-intent study contrasting victim and perpetrator perspectives. All responses are manually annotated and use a fixed GPT-5 judge to measure empathy, validation, exploration, safety, harmfulness and enabling behavior. We quantify prompt-induced changes through paired comparisons, attack-success rates and failure-oriented metrics. This work provides an initial framework for evaluating the robustness of supportive and safe behavior in small LLMs under different forms of prompt manipulation.

More from EMNLP 2026

All EMNLP 2026 papers →