AMD Quark adds online quantization to ROCm for vLLM, ATOM, SGLang serving engines

AMD has introduced online quantization to its Quark library for ROCm, allowing large language model serving engines to quantize weights at load time rather than through a separate offline export step. The feature integrates with vLLM, ATOM, and SGLang, converting high-precision checkpoints to formats like FP8 or MXFP4 layer-by-layer as weights stream into memory. This eliminates the need for calibration data, accuracy-recovery algorithms, and intermediate disk artifacts that offline quantization requires.

The technique targets specific scenarios where offline quantization proves impractical: reinforcement learning rollouts where policy weights change every step, requantization between formats such as NVFP4-to-MXFP4 without disk conversion, rapid mixed-precision scheme exploration through config edits rather than full export runs, and minor modifications to existing quantized checkpoints. AMD reports that online quantization achieves throughput parity with native offline-quantized checkpoints on models including DeepSeek-R1 and Qwen3-235B, though it uses round-to-nearest quantization without the accuracy-recovery techniques available offline.

How online quantization works in ROCm serving stacks

• Layer-wise streaming conversion: Quantization occurs inside each weight-loading hook as tensors arrive, dequantizing to high precision and requantizing to the target format (FP8, MXFP4, NVFP4) one layer at a time rather than processing the entire checkpoint in memory.
• Memory bounding: Processing weights incrementally during I/O keeps peak host and device memory constrained compared to loading a full high-precision checkpoint into memory before conversion, though still higher than loading an already-quantized artifact.
• Round-to-nearest limitation: Online quantization does not support calibration data or accuracy-recovery algorithms such as rotation, SmoothQuant, GPTQ, or AWQ; it relies solely on RTN, trading accuracy optimization for pipeline simplicity and iteration speed.
• Engine integrations: AMD Quark online quantization ships with adapters for vLLM, ATOM, and SGLang serving engines, allowing deployment without modifying the underlying checkpoint storage format or maintaining multiple quantized artifacts per model.
• Requantization path: The system can consume already-quantized checkpoints in one format (e.g., NVFP4 block-scale) and convert to another target format (e.g., MXFP4) during load, demonstrated on AMD Instinct GPUs without intermediate dequantized disk storage.

Why it matters

For ROCm users and AMD Instinct GPU operators, online quantization removes a significant friction point in LLM serving workflows where rapid format experimentation or dynamic weights are required. The technique particularly benefits reinforcement learning practitioners who previously could not afford per-step checkpoint exports, and reduces storage overhead for organizations maintaining models in multiple quantized formats. However, the absence of calibration-based accuracy recovery means production deployments requiring maximum precision will still need offline pipelines, positioning online quantization as a complementary tool rather than a replacement.