CachyLLama's Disk-Backed KV Cache Cuts Agentic Time-to-First-Token From 143 Seconds to 0.99 — a 144.5x Warm-Start Speedup
CachyLLama, a llama.cpp fork by fewtarius aimed at AMD APUs, iGPUs and handhelds with shared system memory, adds a persistent SSD-backed KV cache with hot/warm/cold tiering. On an Ayaneo Flip KB running Qwen3.6-35B over a 15,700-token prompt, cold TTFT was 143.1 seconds versus 0.99 seconds warm — 144.5x. It also adds system-prompt caching across conversations, hybrid MoE checkpoint restoration, MoE expert-activation tracking, per-user concurrency caps, and Vulkan tuning for Strix Halo. MIT-licensed, 101 stars. This directly targets the pathology of local agent loops — re-evaluating thousands of tokens of system prompt and tool definitions on hardware that generates 5–20 tok/s — and is the same problem claude-thermos solves at the API layer, arrived at from the opposite end of the stack.
↳ Follow the thread