Eighteen models got 57% of real finance questions wrong, and 88% of the advanced ones.

Fintech firm Saturn ran 121 real financial questions past 18 models including ChatGPT, Claude, Copilot, Grok and Gemini, repeating each five times for consistency, and measured 43% average accuracy, reported by the Financial Times. Accuracy collapses with difficulty: 88% of responses to advanced queries contained errors, some models failing 99% of the hardest. Errors included arithmetic mistakes, omitted risk warnings, missed tax changes and invented rules. Free tiers wrong 63% of the time against 49% paid; best single model was Claude Opus 5 in reasoning mode at a 39% error rate. This runs at the exact moment agentic finance products are shipping into banks.