MemBoost: Memory-Boosted Framework Slashes LLM Inference Costs via Semantic Answer Reuse and Smart Routing
arXiv·high signal
MemBoost introduces an Associative Memory Engine that lets a cheap model reuse previously generated answers and only escalates uncertain queries to an expensive model. Unlike standard RAG, this supports continual memory growth and cost-aware routing for interactive workloads. Experiments show substantial reduction in large-model invocations while maintaining answer quality — directly applicable to anyone running multi-tier LLM stacks.