Frontier LLMs Score Under 50% on End-to-End Business Intelligence Questions

arXiv 2609.20886·medium signalXBlueskyLinkedInCopy link

BI-Bench harvests real BI projects from public sources and extracts question/ground-truth-answer pairs from actual user dashboards, testing whether models can identify relevant tables, transform data, build joins and answer the business question without manual preparation. Even frontier LLMs score under 50% accuracy. The authors' tool-augmented BI-Agent decomposes the workflow into search, join and transform subtasks orchestrated across specialized data-management methods, and they add a post-training framework that synthesizes trajectories from real BI projects for SFT and RL.