Huawei's new open source technique shrinks LLMs to make them run on less powerful, less expensive hardware

Huaweis Computing Systems Lab in Zurich has developed SINQ (Sinkhorn-Normalized Quantization), an open-source quantization method that significantly reduces the memory footprint of large language models (LLMs) by 6070% without compromising output quality. This calibration-free, fast technique can be seamlessly integrated into existing workflows, enabling models that previously demanded over 60 GB of memory to operate on much more affordable hardware, such as a single Nvidia GeForce RTX 4090, instead of high-end enterprise GPUs like the A100 or H100. The implications of SINQ are substantial, as it