Use a quantized 3B local model as the router and escalate only hard queries to the cloud

Lushbinary·low signal

The 2026 production pattern for cost/latency is a hybrid stack: a small (~3B) on-device model handles routing, classification, and short-form chat, escalating only complex or novel queries to a cloud LLM — delivering 5–10x lower end-to-end latency by eliminating network round-trips for the common case. 4-bit quantization shrinks the model 2–4x while retaining roughly 90–97% of accuracy, making a single-GPU or on-device router viable. The actionable move: measure what fraction of your agent's calls are trivial classification/routing and offload those to a local SLM before paying for frontier inference.