llama.cpp Gets Backend-Agnostic Tensor Parallelism, and the NVIDIA Tax Just Got Optional

PR #19378 landed in llama.cpp this week, and I think most people are underselling what it means. Backend-agnostic tensor parallelism via --split-mode tensor makes multi-GPU inference work across AMD, Intel, and Apple Silicon. Not just CUDA. Everything.

For context: llama.cpp has had a "split mode row" for about 2.5 years, but it was CUDA-only and limited in how it distributed work. The new implementation splits tensors along any dimension using AllReduce operations and works across any backend that llama.cpp supports. If you've got two AMD cards, two Intel Arc GPUs, or even a Mac Studio with multiple chips, you can now run tensor-parallel inference.

This matters because of what else happened this week. A developer published full methodology showing Qwen3.5-122B running at 198 tokens/second on 2x RTX PRO 6000 Blackwell cards. Meanwhile, the r/LocalLLaMA community has converged on Qwen 3.5 27B at IQ3 quants as the consensus pick for 16GB VRAM cards, fitting ~32K context. The models are ready. The inference stack just caught up.

I've been running local models for over a year now, and the single-GPU era for serious work is ending. Two mid-range GPUs with tensor parallelism will outperform one expensive GPU in almost every scenario that matters. The math is simple: memory bandwidth scales linearly, and that's the bottleneck for inference.

The real story isn't performance though. It's vendor independence. AMD's PACE framework also dropped this week, hitting ~380 tokens/sec on Llama 3.1 8B using CPU-only inference on EPYC processors. Between llama.cpp's backend-agnostic TP and AMD's CPU optimization push, the assumption that you need NVIDIA for serious local inference is becoming outdated.

Builders should plan for 2+ GPU setups as the default configuration for local inference in 2026. If you're speccing hardware, prioritize total VRAM over single-GPU speed.