Bringing Nunchaku 4-bit Diffusion Inference to Diffusers

This means that they store the weights in low precision and dequantize them back to high precision at compute time. This reduces memory usage significantly, but it usually does not make inference faster, and can even add a small latency overhead. SVDQuant , the quantization method behind the popular Nunchaku inference engine, takes a different approach.

It runs the main transformer layers with 4-bit weights and activations (W4A4), reducing memory while also speeding up the denoising loop. The details are covered below, but until now, using these checkpoints required a separate inference library. With current Diffusers, loading a Nunchaku checkpoint is as simple as calling , with no local CUDA compilation required thanks to the package.

In addition, the companion diffuse-compressor toolkit lets you quantize new architectures yourself and publish them as regular Diffusers repositories. Table of Contents Getting started with Nunchaku Lite Background: SVDQuant and Nunchaku Introducing Nunchaku Lite Native loading in Diffusers Getting more speed and lower memory Benchmarks Quantizing your own model Ready-to-use checkpoints Conclusion Acknowledgements Getting started with Nunchaku Lite First, install the requirements. You need a recent version of Diffusers and the Hugging Face package: Then load a pre-quantized pipeline like any other Diffusers model: No custom pipeline class or separate inference engine is needed, and there is nothing to compile locally.