GroupMemBench: Best LLM Agent Memory Systems Hit Only 46% Accuracy in Multi-Party Conversations — BM25 Baseline Matches Most

arXiv·medium signal

A new benchmark (arXiv 2605.14498, May 14) reveals that LLM agent memory systems designed for single-user setups fail in multi-party group conversations. The strongest system achieved only 46.0% average accuracy, with knowledge update at 27.1% and term ambiguity at 37.7%. Most strikingly, a simple BM25 baseline matched or exceeded most agent memory systems, indicating the multi-user memory problem remains largely unsolved.