Kog AI Achieves 3,000 Tokens/s Per Request on Standard GPUs — Single Persistent Kernel, No Speculative Decoding

Kog AI / Hacker News·medium signal

Kog AI's inference engine tech preview achieves 3,000 output tokens/s on 8x AMD MI300X and 2,100 on 8x NVIDIA H200 — FP16 with no speculative decoding. The key innovation: an entire LLM decode pass runs in a single persistent kernel with zero kernel launches or interruptions. Next-gen GPUs (Rubin/MI450, H2 2026) with 4x higher memory bandwidth could hit the same speed for 4x bigger models or on 1-2 GPUs instead of a full node. 80 points, 41 comments on HN.