Tip: Increase ubatch Size for 2-3x Faster Prompt Processing on Partially GPU-Offloaded MoE Models

r/LocalLLaMA·medium signal

A builder on r/LocalLLaMA discovered that increasing the physical micro-batch (ubatch) size dramatically improves prompt processing speed for partially offloaded Mixture-of-Experts models. Testing with gpt-oss-120b-F16 on an RTX 3090 (24GB), increasing ubatch from default values yielded 2-3x faster prompt ingestion with 80K context. The improvement comes from better GPU utilization during the prefill phase when expert routing happens. This is directly actionable for anyone running local MoE models for coding with limited VRAM.