Skip to content
conftrace_
2026 EMNLP EMNLP 2026

REMP: A Structured Event-Probing Benchmark for Long-Horizon Role-Playing Dialogue

AbstractLarge language models (LLMs) are increasingly used in role-playing applications, yet evaluating whether they maintain character-consistent behavior over extended interactions remains challenging. Existing benchmarks mostly focus on short exchanges, static long-context inputs, or isolated aspects of role-playing, making long-horizon failures difficult to diagnose. We present REMP (Relationship-Emotion-Memory-Persona), a structured event-probing benchmark for long-horizon role-playing dialogue. REMP constructs multi-chapter dialogue trajectories from character profiles and embeds diagnostic probe events targeting persona consistency, emotional consistency, memory integrity, and relational continuity. We further introduce REMP-EVAL, an anchor-based multi-turn evaluation protocol that assesses model behavior after event triggers using dimension-specific references. The current release contains 160 characters organized into 80 dialogue trajectories, each averaging about 60 sessions and 800 turns. Experiments on 11 LLMs show that memory integrity and emotional consistency are the main bottlenecks, indicating that long-horizon role-playing requires not only long-context retrieval but also sustained affective and relational consistency.

🌉 Interdisciplinary Bridge - Artificial Intelligence and Natural Language Processing

More from EMNLP 2026

All EMNLP 2026 papers →