LLM Latency Is Long Enough to Hide an Online Learner's Gradient Step, Worth Roughly 8x in Production

arXiv 2608.27244·medium signalXBlueskyLinkedInCopy link

In semantic data processing systems LLM compute is 80 to 90% of query cost and a single call costs 10^5 to 10^7 times a relational predicate, which inverts the classical adaptive-query-processing constraint that online learners must stay lightweight. At LLM latency, per-call gradient steps and per-batch threshold solves fit entirely inside the round-trip, so the learner is effectively free. A production case study in Snowflake's Cortex AISQL composes a memoization layer, an online per-call filter-ordering learner and an online per-batch cascade-routing learner, which compose multiplicatively to an 11.4x upper bound on a conjunction-filter workload, reduced to about 8x once self-selection, sample-budget shrinkage and selectivity drift are accounted for.