One MI300X GPU Runs DeepSeek V4 Flash At 168.6 Tok/S

One MI300X GPU runs DeepSeek V4 Flash's 304 billion parameters unquantized, hitting 168.6 tokens per second, per a single-author GitHub repo documenting the setup.

Running a 304-billion-parameter model unquantized normally means splitting weights across several GPUs. This claims one card, holding 156.67 GB, with prefill up to 8.5K tokens per second and no out-of-memory error at a 64-stream burst. If the numbers hold, teams weighing AMD MI300X for large-model inference get a concrete data point to test against their own workloads.

The fix behind it matters more than the throughput number. MI300X uses AMD's FNUZ FP8 variant, not the OCP standard vLLM's FP8 path assumes. The repo corrects that with a cache-writer overlay selecting float8e4b8 and setting FP8_MAX to 224.0 to match ROCm's expectations. A second change, custom AITER GEMM tuning for the gfx942 architecture, adds 42 to 62 percent to decode speed on top of the format fix.

Yes, but: this is one commit from one author, 18 stars, Apache-2.0 licensed. There's no independent benchmark confirming the numbers.

The FP8 fix is the more durable finding than the tok/s number. It's a correctness patch, not a speed trick, and any team running FP8 inference on MI300X runs into the same OCP-versus-FNUZ mismatch eventually. Whether 168.6 tok/s holds up outside one author's repo, or gets folded into vLLM's own AMD support, is what to watch. It surfaced in the August 4 roundup of AMD inference work, with no vendor confirmation or wider deployment reported yet.