Skip to content
conftrace_
2026 EMNLP EMNLP 2026

COLLABMEM: A Diagnostic Benchmark for Collaborative Memory Operations in Multi-Agent LLM Systems

AbstractMulti-agent LLM systems increasingly rely on persistent memory to coordinate role-specialized work. Correctness requires more than recalling a fact: the fact must reach the right role, remain hidden from unauthorized roles, carry the right source, and update when a more authoritative value appears. We introduce COLLABMEM, a diagnostic benchmark for collaborative memory operations. Its matched memory-only harness fixes the trace, target role, answerer, judge, and gold event, localizing errors to sharing, access, routing, provenance, conflict resolution, or retention. COLLABMEM contains 58 hand-authored scenarios (1,118 core probes), a 174-scenario scaled split (3,359 probes), adversarial and compositional hard sets, and six transactive-memory competencies. We also propose TMS-GRAPH++, a scope-aware graph memory framework with role-filtered views, writer/provenance structure, and temporal [CURRENT] marking. Across five evaluation splits, TMS-GRAPH++ achieves the highest non-Oracle average score (69.6%). Evaluations using the official SDKs of Mem0, Letta, and A-Mem show that shared archives help, but reliable collaborative memory still requires explicit visibility, provenance, authority, and retention policies. We will release the dataset, evaluation code, and reproduced outputs to support future work on multi-agent memory systems.

More from EMNLP 2026

All EMNLP 2026 papers →