GSQ: Gumbel-Softmax Sampling Closes Gap Between Simple and Vector Quantization for LLMs at 2-3 Bits

arXiv·medium signal

GSQ introduces a Gumbel-Softmax sampling approach to scalar quantization that matches the accuracy of complex vector/trellis methods (QTIP, GPTVQ, AQLM) while retaining the simplicity and deployment compatibility of GPTQ/AWQ. Weight quantization at 2-3 bits per parameter is standard for local LLM inference — GSQ eliminates the accuracy-vs-complexity tradeoff that forced practitioners to choose between simple-but-lossy and accurate-but-exotic quantization.