Fetching from the wire…
01
02
03
04
05
06
07
08
Infra2026-09-21 · source-backed
PR #40201 removes CUDA context creation from import sglang: get_device_sm() answers from NVML while torch.cuda is uninitialized, and parser CLI choices come from dependency-free name lists instead of pulling registries worth about three seconds of imports into every argument-parsing process. A process holding a CUDA context can't be a fork() parent, which is what blocked forkserver worker startup. Measured on Qwen3-30B-A3B on an H200: serve-to-health drops from 47.3s to 39.5s single-GPU, 56.0s to 40.5s on DP2, 46.6s to 31.4s on EP2 with DeepEP low-latency.
Each link below shares sources, entities, or timing with this story.
The September 5 release adds beam search via a beam_width request parameter returning the n best sequences, though it doesn't yet combine with speculative decoding, disaggregation, DP attention or HiCache. DeepEP v2's fixed-size ElasticBuffer engine as --moe-a2a-backend deepep...
The file is 1.56TB. That's the first thing you notice about the moonshotai/Kimi-K3 Hugging Face repo that went live today. 2.8 trillion total parameters, 104B activated, 896 routed experts with 16 selected plus 2 shared per token, 93 layers split 69 Kimi Delta Attention and 24...
PR #19378 landed in llama.cpp this week, and I think most people are underselling what it means. Backend-agnostic tensor parallelism via --split-mode tensor makes multi-GPU inference work across AMD, Intel, and Apple Silicon. Not just CUDA. Everything. For context: llama.cpp h...
The day-0 SGLang post dated August 26 gives the architecture the release megathread didn't: 125B main parameters plus a separate 51.2B Per-Layer Embedding table at about 95.4 GiB in BF16, 6B activated per token, 48 layers split 36 GDN linear attention and 12 QSA sparse attenti...
Wafer.ai ran K3 at TP8 on a single MI355X node: 952 tok/s aggregate, 118 tok/s single-stream decode, ~13k tok/s steady-state prefill after fixing missing PyTorch sampling functions and AITER MLA prefill kernels via SGLang and ROCm. Against a two-node TP16 B200 deployment that'...
Speculative decoding (a small draft model proposes tokens the target verifies in parallel) gives 2-5x latency wins, but only in memory-bound, low-batch regimes. At large batch sizes the GPU is already compute-bound, and the extra draft-and-verify work makes inference *slower*...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.