Skip to content
conftrace_
2026 EMNLP EMNLP 2026

VIBE-Bench: Evaluating Personalized Large Language Models When Profiles Don’t Mean Preferences

AbstractPersonalized Large Language Models (PLLMs) aim to tailor responses to individual users, where a central challenge is preference reasoning: inferring query-relevant preferences from user-related history. Existing benchmarks, however, largely assume that such preference can be retrieved from semantically related history. We study an underexplored but practically important regime, profile-preference conceptual misalignment (PRCM), where observable profile cues and query-specific preferences lie in different concept spaces, making semantic retrieval inconsistent for personalization. We introduce Vibe-Bench, a benchmark with two psychology-grounded tasks, 3,504 personas and 12,239 dialogues, including a manually verified gold test set, and requires cross-concept preference reasoning beyond surface semantic overlap. Experiments with several personalization methods show that current PLLMs largely rely on shallow semantic correlations and fail to acquire robust cross-concept mappings. These findings establish PRCM as a distinct failure regime in PLLMs and position Vibe-Bench as a focused testbed for advancing preference reasoning beyond semantic matching.

More from EMNLP 2026

All EMNLP 2026 papers →