Measure whether your agent's memory actually got used, not just whether scores went up — same headline gain, wildly different save/retrieve/update evidence

arXiv 2608.04003·medium signalXBlueskyLinkedInCopy link

PAST-Bench runs personal agents through ordered fresh-session task sequences under matched conditions that toggle retained experience on and off, across 26 scenarios and 204 episodes, seven base models, and four agent frameworks. Improvement from memory is real but uneven, and the central methodological finding is that two agents posting the same later-task gain can differ sharply in whether that gain is actually supported by evidence of the intended save → retrieve → update pathway — meaning aggregate benchmark lift is not a valid proxy for a working memory layer. Their Hermes+ variant adds five targeted interventions across the agent loop and improves most on tasks requiring outdated state to be replaced, which is precisely the failure mode append-only memory stores hide.