Tools
vLLM's disaggregated encoder proxy batches images per request for 6.8% more throughput
PR #57095, merged 2026-09-17T10:45Z, changes the vLLM EPD proxy to group images from the same user request assigned to the same encoder instance into one multi-image HTTP request, with item positions returned in EC metadata so results map back to the original images including repeats and derived hashes. A same-node A/B on Qwen3.5-35B-A3B BF16, 4E+4PD across two GB200 nodes over Mooncake RDMA at concurrency 96 measured mean throughput 90.22 to 96.34 req/s (+6.8%), mean latency 1035.7 to 975.2 ms, P99 3036.0 to 2862.8 ms, and encoder subrequests per round falling 4,924 to 3,878 (-21.2%). Audio and video remain unbatched and no new configuration is added.
Source
↳ Follow the thread