mzCache speeds up on-device LLMs after phones dump their memory

Phones run many apps at once, so the OS evicts whatever memory it thinks it can safely reload later. For a large language model running on-device, that eviction is expensive: the next prompt either waits on a slow storage read or forces the model to recompute its entire key-value cache from zero.

That delay is the gap between an on-device assistant feeling instant and feeling broken. A voice assistant or in-app copilot that stalls for a couple of seconds after you switch apps trains people to stop trusting it.

A new paper describes mzCache, a memory system built to make that recovery cheap. Rather than treating a model's weights and KV cache as one block to evict or keep, it splits them into fine-grained shared buffers, so the OS can evict and restore pieces of memory instead of all of it. It also uses the unified memory on mobile chips to run inference on the GPU while the CPU restores evicted data at the same time, according to the paper on arXiv.

Time-to-first-token falls 2.1 to 5.5 times once an eviction happens. The authors didn't stop at a benchmark; they built mzCache into a working Android app, which is the harder claim, since a lot of inference-speedup research never leaves a simulator.

The paper doesn't say how mzCache compares to Apple's or Google's newest on-device runtimes, or whether the fine-grained buffers add overhead when memory pressure never hits. Building local-first AI features for mobile means asking whether your memory layout survives the OS deciding it wants that RAM back. Model size is a secondary concern.