MiniCache Turns Program-of-Thought Programs Into Parameterized Cache Objects for 3.1× Lower Latency

arXiv 2607.20507·medium signal

Chen, Piao, Feng, Hu, and Li from Tsinghua convert Program-of-Thought programs into parameterized cache objects reusable across structurally similar requests, with one small model doing double duty — semantic variable extraction on cache hits and speculative drafting during target-LLM generation. On WebShop, Formula, and CodeTAT-QA they report up to 3.1× lower latency and 2.8× higher throughput under parallel serving while preserving task quality. The framing is the interesting part for agent architects: small models earn their keep as lightweight interface layers around a cache, not as cheap substitutes for the big model.