Attention backend swap flips one in five tokens

Swapping only the attention kernel flips 15 to 20% of a local model's top-1 token picks late in a 96,000-token prompt, per a Level1Techs teardown. That matters for anyone running agents on local hardware, since tool calls depend on the model reliably picking the same structured tokens. The teardown captured full-vocabulary logits and computed KL divergence in FP64 against a reference model. It tested Qwen 3.6-27B on an RTX PRO 6000, swapping FlashAttention 2, Flash Inference and Triton on identical BF16 weights. It hit 384 points and 144 comments on Hacker News.

Quantization made it worse. NVFP4 weight quantization pushed disagreement to roughly 50% by 88,000 tokens of context. NVFP4 and AWQ W4A16 both failed tool-call sequences that BF16, FP8 and INT8 W8A16 completed. INT4 KV cache caused tool-call failures at 100,000 context that didn't recover, where INT8 KV cache degraded but stayed usable.

I've run local models for agent work all year and hit this exact pattern. Fine for 20,000 tokens, then somewhere past 60,000 the tool calls start coming back malformed, and I blamed my harness code. This teardown says the numerics were drifting, not my prompt logic.

The fixes this points to

Skip W4 weight quantization if an agent depends on tool calls, since FP8 and INT8 W8A16 held where NVFP4 and AWQ W4A16 didn't. Skip INT4 KV cache for long agent sessions, since that failure mode doesn't recover once it starts. And a benchmark run on one attention backend doesn't transfer to a deployment on another, per the kernel-swap result itself.

A related benchmark posted the same week on r/LocalLLaMA measured a llama.cpp PR for DFlash 2 speculative decoding on Qwen. It found a 2.26x speedup, 67.97 to 153.91 tokens per second, for 2.7 GB more VRAM. That poster also found two flags doing nothing on the current code path. --spec-draft-p-min is never read, and --spec-draft-n-max 7 sits past the point of diminishing returns. Speed work on local inference is loud right now. This is the first teardown that measured the feels-dumber complaint instead of just describing it.