AgentServe: Algorithm-System Co-Design for Efficient Agentic AI Serving on Consumer-Grade GPUs

arXiv 2603.10342·medium signalXBlueskyLinkedInCopy link

Agent inference patterns — short reasoning-action loops with interleaved tool calls — create fundamentally different serving challenges than standard LLM inference: bursty compute, variable latency, and priority inversion between active and waiting agent states. AgentServe co-designs adaptive speculative decoding tuned for tool-call token patterns with a priority-aware request scheduler, achieving viable agent serving on consumer-grade GPUs. Practical for builders who cannot afford cloud-scale inference infrastructure.