mzCache Cuts On-Device LLM Time-to-First-Token 2.1-5.5x When the OS Evicts Model Memory

arXiv 2609.01338·medium signalXBlueskyLinkedInCopy link

Mobile devices multitask, so the OS evicts LLM weights and KV cache under memory pressure, forcing slow storage reads or full KV recomputation when the next request arrives. mzCache partitions LLM memory into fine-grained shared buffers so eviction and restoration can be partial and concurrent across processors, then uses the unified memory of mobile SoCs to run inference on the GPU while the CPU restores in parallel, with hybrid swap and backward-out eviction policies to keep restoration fast from any eviction state. Deployed as an Android application, it achieves a 2.1 to 5.5x reduction in time-to-first-token against storage-backed partial offload.