llama.cpp Merges Backend-Agnostic Tensor Parallelism — Multi-GPU Inference Without CUDA Requirement

GitHub·high signal

PR #19378 adds backend-agnostic tensor parallelism to llama.cpp via --split-mode tensor. Unlike the legacy 'split mode row' that existed for 2.5 years, this implementation splits tensors along any dimension using AllReduce operations and works across any backend — AMD, Intel, Apple Silicon, not just CUDA. This is the most significant multi-GPU local inference improvement in llama.cpp's history, making fast multi-GPU setups accessible to non-NVIDIA hardware for the first time.