A ReAct agent averaging 77% per run succeeds all five times on only 53% of tasks.

On AppWorld with GPT-4.1, that 24-point shortfall is what the authors call the consistency gap. Their framework adds a Consistency Analyzer pinpointing where a trajectory is likely to flip and a Guideline Generator committing the diagnosis to episodic memory, raising all-five-run success by 16 points on same-task evaluation and 13 on similar-task generalization. arXiv 2609.08832 If you report agent reliability as a per-run average, you're reporting a number your users don't experience.