LLM-synthesized CodeQL queries beat baseline query suites by 82% F1 and cost far less than scanning repos with the model directly

arXiv 2609.10412·high signalXBlueskyLinkedInCopy link

An empirical study generated CodeQL queries from National Vulnerability Database entries and evaluated them across real-world vulnerabilities, finding an 82% average F1 improvement over baseline CodeQL queries. The accompanying cost-benefit analysis found direct LLM scanning of whole repositories computationally and financially prohibitive, while using the LLM once to synthesize a reusable static-analysis query is cheap and scales. The pattern generalizes: spend model tokens producing a deterministic artifact you run many times, rather than putting the model in the hot path of every scan.