NVIDIA's TensorRT Model Optimizer Brings QAT and QAD Techniques to Recover AI Accuracy Lost in Quantization

NVIDIA's TensorRT Model Optimizer introduces QAT and QAD techniques that recover AI accuracy lost during quantization by simulating low-precision arithmetic during training, with QAD leveraging a full-precision teacher model to guide quantized models, delivering benchmark improvements on models like Llama Nemotron Super via PyTorch and Hugging Face-compatible APIs.

• Quantization aware training (QAT) and quantization aware distillation (QAD) are emerging techniques that help AI models recover accuracy lost during post-training quantization (PTQ) by simulating low-precision arithmetic during an additional training phase.
• QAD combines QAT with knowledge distillation, using a full-precision teacher model to guide a quantized student model, resulting in superior accuracy recovery compared to PTQ alone, as demonstrated by significant benchmark improvements on models like Llama Nemotron Super.
• NVIDIA's TensorRT Model Optimizer provides accessible APIs compatible with PyTorch and Hugging Face to apply QAT and QAD workflows, supporting key formats such as NVFP4 and MXFP4 for efficient low-precision deployment on NVIDIA GPUs.