Diffusion models quantization benchmark

Diffusion models became de facto standard in image generation. While remarkably powerful, they rely on a multi-step process of iterative denoising, which takes time. Text generation is also processed in a sequential manner, however the text can be streamed and visualized token by token, giving a natural, real-time feeling, similar to how humans read or write a text. With images, we must wait for the final denoising step to complete before a usable result appears. This makes optimizing diffusion models for speed and efficiency a critical challenge.

Diffusion models can be broadly categorized by their underlying architecture:

• Pure convolutional diffusion models
• Hybrid (convolution + transformer) diffusion models
• Fully transformer-based diffusion models

This architectural distinction has a direct impact on which optimization strategies are most effective.Transformer-based diffusion models benefit greatly from quantization - especially fp8 and nf4 on newer NVIDIA GPUs with native hardware acceleration. Convolutional models, by contrast, respond better to pruning and typically demand less VRAM. Regardless of architecture, reducing the number of denoising steps remains a universal approach for faster generation.

This article presents a benchmark for comparing state-of-the-art text-to-image diffusion models across four dimensions: generation speed, image quality, VRAM consumption, and quantization efficiency.

Models compared

For the purpose of this benchmark, multiple models were selected to span the full design space of modern text-to-image diffusion models: from the lightweight and widely-deployed SD 1.5, through the SDXL family and its distilled variants (Turbo, Lightning, Hyper, LCM), to the latest transformer-based Flux.1 architecture. Step counts range from 1 to 30, covering both standard and accelerated inference regimes.

Table 1. Datasheet of benchmarked models.

Comparison dimensions

• Speed - total generation time, time per denoising step, step count
• Quality - CLIP score (text alignment), Laplacian sharpness, colorfulness (Hasler & Süsstrunk), Shannon entropy
• VRAM - model weight footprint and peak generation VRAM
• Quantization - FP16 / BF16 / INT8 / NF4 / INT4 / INT4-GEMM / FP8-W / FP8 (bitsandbytes + torchao), FP8_static

Quantization scope: weights-only vs weights + compute

Not all quantization formats speed up inference. The key distinction is whether the GPU executes matrix multiplications in a lower-precision format, or just stores weights compactly and quantized back to fp16/bf16 before each operation. Weights-only approach reduces the VRAM stress allowing for inference on devices with less VRAM or larger batchsize (throughput) on the same device. Weights + compute allows for both - faster inference and larger batchsize, however require newer GPU architectures.

Table 2. Comparison of quantisation modes and supported hardware.

When quantization speeds things up

By leveraging native Tensor Core instructions, int4_gemm, fp8, and fp8_static perform matrix multiplication directly on quantized values and fuse the dequantize directly into the multiply, eliminating the overhead (weights are still dequantized before activations in the model). int4_gemm uses symmetric INT4 and runs on any Ampere+ GPU. fp8 uses FP8 dynamic activation quantization (rescales activations every step) and fp8_static uses pre-calibrated frozen scales (no per-step overhead) - both require Ada Lovelace or Blackwell (RTX 4090+).

int8 vs fp8_w - both 8-bit weight-only, different number formats

Both store weights in 8 bits and dequantize before compute, so neither speeds up inference. The difference is what those 8 bits represent:

• int8 stores weights as 8-bit integers: a uniform linear grid from -128 to 127. Each block of weights gets one scale factor. The grid spacing is constant, which is a poor fit for neural network weights that cluster near zero with rare large outliers. Before compute weights are dequantized to FP16.
• fp8_w stores weights as 8-bit floating point (E4M3: 4 exponent bits, 3 mantissa bits). Because it has an exponent, the grid is non-uniform - denser near zero, sparser at large magnitudes. This matches the distribution of trained weights much better. Dequantizes to bf16.

fp8_w should give marginally better quality than int8 at the same VRAM tier, for the same reason NF4 outperforms uniform int4: floating point naturally matches the bell-curve distribution of neural network weights.

Results

All benchmarks were run on an RTX PRO 6000 Blackwell Server Edition (96 GB GDDR7). This matters for interpreting the quantization results: fp8 and fp8_static speedups require Ada Lovelace or newer (RTX 4090+), and int4_gemm requires Ampere or newer (RTX 3090+). Results for weights-only formats - nf4, int4, int8, fp8_w - generalize to any CUDA GPU, since they affect VRAM and quality but not compute throughput.

Table 3. Results obtained for RTX PRO 6000 Blackwell Server Edition (96 GB GDDR7)

Python 3.11 · PyTorch 2.11 · diffusers 0.38 · 13 prompts · 3 seeds each (SD 1.5, SDXL Turbo, SDXL, LCM-SDXL, Flux.1 Schnell, Flux.1 Dev, SDXL Lightning 2-step & 4-step, Hyper-SDXL 4-step).

Why fp8 and fp8_w are slower on UNet models (Stable Diffusion family) and similarly fast to fp16 on DiT models (Flux)

FP8 tensor cores only accelerate matrix multiplies (GEMM). The two architectures have fundamentally different operation profiles:

• UNet (SD family): the backbone is dominated by Conv2d layers, not linear projections. fp8 quantized the linear layers inside attention blocks, but the Conv2d operations, which do the majority of the compute, are untouched and stay in bf16. Additionally, Float8DynamicActivationFloat8WeightConfig, used for quantization, measures and scales activations dynamically on every forward pass, adding per-layer overhead to every layer, including the Conv2d ones that gain nothing. 
• DiT (Flux): a pure transformer with no convolutions - every operation is attention + linear projections. FP8 quantizes essentially 100% of the computer, and FP8 GEMM tensor cores accelerate it all. FP8 tensor cores do run at ~2× fp16 throughput on the GEMM itself, but the dynamic activation quantization overhead consumes most of that gain. Net result: roughly breakeven, with overhead tipping the balance slightly negative (~2.5% slower). Float8DynamicActivationFloat8WeightConfig recomputes activation scales on every forward pass. For each linear layer it:

1. Scans activations to find amax(|x|) - one extra read over the activation tensor
2. Multiplies activations by the derived scale - one extra write
3. Runs the FP8 GEMM
4. Dequantizes the output back to bf16

Steps 1-2 are pure overhead that fp16 does not pay. The 2× GEMM speedup must first absorb this cost before any net gain appears.

What actually unlocks fp8 speedup for transformer-based models: fp8_static

The dynamic activation quantization overhead is the key problem. Calculating the required scale factors on the fly introduces a runtime overhead. fp8_static solves this with an offline calibration step. Because FP8 has a limited dynamic range, precision-sensitive operations - such as LayerNorm, softmax, and residual additions must stay in bf16 to prevent mathematical instability. Only the linear projections run in fp8. In regular fp8 quantization, the scale factor used for dequantization to fp16 is recomputed for each layer for each inference run. It creates compute overhead that stalls compute-bound hardware (data arrives fast enough but the arithmetic units are saturated). Fp8_static calculates those scale factors in the calibration run and reuses them in runtime.

Table 4. Fp8 vs fp8_static. When to use which one.

Speeding up conv-based UNets beyond quantization

Benchmark results presented in the table above prove that quantization does not speed up convolutional-based diffusion models. Quantization targets nn.Linear layers: the attention projections and feed-forward blocks. In SD-family UNets, the dominant compute is in ResNet blocks (Conv2d + GroupNorm + SiLU), which are untouched by every quantization format evaluated here. Therefore other techniques should be used to boost the inference speed for SD models. 

• torch.compile generates optimized CUDA kernels that fuse Conv2d + GroupNorm + SiLU into a single kernel dispatch, eliminating Python-level overhead between operations. On SD, this typically yields 20-40% wall-clock reduction at no quality or VRAM cost.
• Channels-last memory format. PyTorch stores tensors in NCHW format by default. cuDNN's Conv2d kernels are faster in NHWC (channels-last) because GPU memory access patterns align better with the convolution sliding window.
• TensorRT with INT8 calibration. TensorRT's PTQ calibration can quantize Conv2d layers to INT8, unlike torchao, which is limited to nn.Linear. This is the only quantization approach that delivers real throughput gains on conv-dominated models. The trade-off is setup complexity: ONNX export, calibration passes, and GPU-architecture-specific engine compilation that cannot be reused across GPU generations.

Visualisations

The following samples illustrate the results obtained for different quantisation levels and strategies.

Conclusions

The most universal and effective optimization is reducing the number of denoising iterations. Some models are designed to produce high-quality results in as few as 4 steps, making this the single biggest win available regardless of architecture. 

Quantization consistently reduces VRAM footprint across all model types, enabling deployment on resource-constrained devices or improving throughput at scale. Its effect on speed, however, is architecture-dependent: for transformer-based models, it accelerates inference only on newer GPU architectures with native hardware support, while for convolutional models, it can actually add processing overhead.

For convolutional diffusion models specifically, the most effective optimization strategies are kernel fusion, channel-last memory format, pruning, and TensorRT INT8 calibration.

Reviewed by: Michał Zaręba