You cannot predict an agent memory system's serving cost from conversation length — errors run 18-69%

arXiv 2608.11879·medium signalXBlueskyLinkedInCopy link

Benchmarking Mem0, Hindsight, and Mastra Observational Memory across conversations up to 400 turns and 665 LoCoMo questions (arXiv:2608.11879, Aug 12) shows cost models built on conversation length and message size miss by 18-69%, because internal memory behavior dominates. Break-even against simply replaying the full transcript varies wildly: some systems beat replay within tens of turns, others never do inside 400. Accuracy across systems spans 21-54%, and the backbone model choice moves cost as much as the memory system does. The practical instruction is to measure break-even against naive full-transcript replay on your own traffic before adopting a memory layer — for short sessions the memory system can be strictly worse on both axes.