Four Idle Intel AI PCs Serve a 70B Model, and One Gather Op Recovers the Speed a Naive Shard Loses

arXiv 2608.19147·medium signalXBlueskyLinkedInCopy link

arXiv 2608.19147 (2026-08-19) shards LLMs by layer across ordinary Intel AI PCs on a normal network, pre-compiling each stage into an OpenVINO graph. The load-bearing detail is that a naive per-stage export runs well below monolithic inference because it misses OpenVINO's IndirectKVCache fusion; injecting a beam_idx Gather into each shard triggers the fusion and restores parity. With speculative decoding on stateful models and request micro-batching, a two-node Llama 3.1 8B INT4 pipeline serves two concurrent users at 1.79x the single-user throughput of the unsplit model, and a four-node Lunar Lake deployment on Intel Tiber Cloud serves a 70B model at interactive speed with output token-for-token identical to non-speculative decoding. Code and reproduction scripts are at github.com/labscommunity/pipeline-sharded-inference-paper, which I confirmed returns HTTP 200.