Skip to content
conftrace_
2026 ICML ICML 2026

Position: We Need Large Language Models Optimized For Our Well-Being

Large language models are useful because we taught them to give us what we want. That works when success can be judged immediately, but people increasingly bring these systems their relationships, their hard decisions, and their long-term goals, where what a user wants to hear and what serves them best are frequently different. We argue that LLM providers should offer at least one widely accessible, opt-in mode optimized and evaluated for long-term well-being rather than next-turn approval. This is a pressing need, as models have been found to endorse questionable framings well above human baselines, users take AI advice readily without their well-being improving, and sycophantic models raise dependence while lowering prosocial intent. We propose three principles—change the objective, give users explicit relational roles, avoid paternalism—and organize the design space around three choices the current paradigm makes implicitly: the horizon over which well-being is measured (When), whose interests it represents (Who), and what role the assistant plays (How).

More from ICML 2026

All ICML 2026 papers →