The 'Memory-Reward Trap': RoMeRL Shows Trajectory-Level Rewards Poison Agent Memory, and Fixing Credit Assignment Cuts Stored Memory 84.4%

arXiv / HuggingFace Daily Papers·medium signalXBlueskyLinkedInCopy link

arXiv 2608.02508 (v3 posted Aug 10) names a failure mode in self-evolving agent memory: when an agent retrieves several memories for one trajectory, the single trajectory-level reward is assigned to all of them, so irrelevant experiences receive misleading utility updates and get reinforced. RoMeRL's reduced-order utility states decouple that credit assignment, and on ALFWorld and LifelongAgentBench it cuts the cold-Q ratio 80.0%, increases feedback density ~6.0x, reduces maintained memory size 84.4%, and drops LLM calls 21.1% while improving task performance. For anyone running a long-lived agent with a growing memory store, this is a concrete diagnosis of why those stores bloat with junk that keeps getting retrieved.