DeepSeek V4 Flash Runs Unquantized on One AMD MI300X at 168.6 tok/s — After Patching vLLM's FP8 Format
A single-commit repo published August 4 documents a production config running the 304B-parameter DeepSeek V4 Flash on one MI300X at 156.67 GB in memory with no quantization: 168.6 tok/s single-stream decode, 7.9–8.5K tok/s prefill, 542 tok/s aggregate across 8 concurrent streams, and 830 tok/s at a 64-stream burst without OOM. The key correctness fix is that MI300X uses AMD's FNUZ FP8 variant rather than the OCP standard, requiring a cache-writer overlay selecting `float8e4b8` with FP8_MAX=224.0 and shuffled write offsets on ROCm. Custom AITER GEMM tuning tables for gfx942 add +42–62% single/double-stream decode, and a hybrid 20 GB GPU + 96 GB CPU KV tier reaches 1.93M-token capacity. Only 18 stars — early, single-author, Apache-2.0.
↳ Follow the thread