MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation
Abstract
AbstractMemory systems for conversational LLMs are conventionally evaluated by direct, fact-seeking questions about prior dialogue (Direct QA): can the model recall fact X from a prior conversation? We tested whether higher Direct QA accuracy correlates with higher user satisfaction in a four-month deployment involving 40 users, 1,872 sessions, and seven memory conditions. Existing-benchmark Direct QA accuracy varies from 19.7% to 70.1% across the seven conditions, but satisfaction does not change. We hypothesize that existing benchmarks and user satisfaction measure different capabilities: benchmarks measure elicited retrieval—recall when explicitly asked—whereas natural conversation requires detecting relevance and appropriately integrating prior context into a response. To examine this distinction, we introduce MemUse, a benchmark of real user-cued memory moments drawn from the deployment and evaluated using integration-aware judgments of natural conversational responses. Holding the model and context fixed, the same system that scores 78.8% on Direct QA references only 7.9% of those facts in conversation, revealing a 71-point retrieval–integration gap. Within a small observational subset of naturally occurring memory moments, successful Natural Integration is positively associated with user satisfaction, whereas Direct QA performance is not. We release the deployment corpus and MemUse together with the associated judgments and scoring prompts.