# AMD Quark adds online quantization to ROCm for vLLM, ATOM, SGLang serving engines

> **Key takeaway:** AMD's Quark quantization library now supports online quantization that converts model weights during loading rather than requiring pre-exported quantized checkpoints, enabling faster iteration for RL training rollouts and mixed-precision exploration on ROCm.

**Published:** 2026-11-05T18:02:35.710019+00:00 
**Source:** AMD ROCm Official Blog 
**Category:** ai-local-edge 
**Canonical URL:** https://fosswire.org/news/amd-quark-adds-online-quantization-to-rocm-for-vllm-atom-sgl.html 

## Summary
AMD has introduced online quantization to its Quark library for ROCm, allowing large language model serving engines to quantize weights at load time rather than through a separate offline export step. The feature integrates with vLLM, ATOM, and SGLang, converting high-precision checkpoints to formats like FP8 or MXFP4 layer-by-layer as weights stream into memory. This eliminates the need for calibration data, accuracy-recovery algorithms, and intermediate disk artifacts that offline quantization requires.

The technique targets specific scenarios where offline quantization proves impractical: reinforcement learning rollouts where policy weights change every step, requantization between formats such as NVFP4-to-MXFP4 without disk conversion, rapid mixed-precision scheme exploration through config edits rather than full export runs, and minor modifications to existing quantized checkpoints. AMD reports that online quantization achieves throughput parity with native offline-quantized checkpoints on models including DeepSeek-R1 and Qwen3-235B, though it uses round-to-nearest quantization without the accuracy-recovery techniques available offline.

## How online quantization works in ROCm serving stacks
- **Layer-wise streaming conversion:** Quantization occurs inside each weight-loading hook as tensors arrive, dequantizing to high precision and requantizing to the target format (FP8, MXFP4, NVFP4) one layer at a time rather than processing the entire checkpoint in memory.
- **Memory bounding:** Processing weights incrementally during I/O keeps peak host and device memory constrained compared to loading a full high-precision checkpoint into memory before conversion, though still higher than loading an already-quantized artifact.
- **Round-to-nearest limitation:** Online quantization does not support calibration data or accuracy-recovery algorithms such as rotation, SmoothQuant, GPTQ, or AWQ; it relies solely on RTN, trading accuracy optimization for pipeline simplicity and iteration speed.
- **Engine integrations:** AMD Quark online quantization ships with adapters for vLLM, ATOM, and SGLang serving engines, allowing deployment without modifying the underlying checkpoint storage format or maintaining multiple quantized artifacts per model.
- **Requantization path:** The system can consume already-quantized checkpoints in one format (e.g., NVFP4 block-scale) and convert to another target format (e.g., MXFP4) during load, demonstrated on AMD Instinct GPUs without intermediate dequantized disk storage.

## Why it matters
For ROCm users and AMD Instinct GPU operators, online quantization removes a significant friction point in LLM serving workflows where rapid format experimentation or dynamic weights are required. The technique particularly benefits reinforcement learning practitioners who previously could not afford per-step checkpoint exports, and reduces storage overhead for organizations maintaining models in multiple quantized formats. However, the absence of calibration-based accuracy recovery means production deployments requiring maximum precision will still need offline pipelines, positioning online quantization as a complementary tool rather than a replacement.