MiniCache turns Program-of-Thought programs into parameterized cache objects for 3.1x lower latency.

Tsinghua's Chen, Piao, Feng, Hu, and Li convert PoT programs into parameterized cache objects reusable across structurally similar requests, with one small model doing double duty: semantic variable extraction on cache hits and speculative drafting during target-LLM generation (arXiv 2607.20507). Up to 3.1x lower latency and 2.8x higher throughput under parallel serving on WebShop, Formula, and CodeTAT-QA. The architectural framing is the takeaway: small models earn their keep as lightweight interface layers around a cache, not as cheap substitutes for the big model.