NVIDIA releases Snapshot open-source tool to accelerate GPU workload restores

NVIDIA has introduced NVIDIA Snapshot, an open-source component of the DSX AI Factory Platform designed to optimize the deployment of large language model (LLM) workloads. The tool addresses the 'cold start' problem in AI inference, where initializing a model—including loading weights, compiling kernels, and capturing CUDA graphs—can take several minutes.

By capturing the fully initialized state of both CPU and GPU memory, Snapshot allows Kubernetes users to restore inference replicas in as little as 3.6 to 42.7 seconds. This mechanism enables much faster scaling and recovery when inference demand spikes or when individual replicas fail in a production environment.

Technical implementation and performance

• Kubernetes Integration: The system utilizes Kubernetes custom resources, allowing users to create 'PodSnapshot' objects for existing workloads or 'SnapshotJob' objects to checkpoint a template immediately upon readiness.
• Multi-GPU Support: Snapshot can manage complex workloads spanning multiple GPUs on a single node, supporting parallelism strategies including tensor parallelism (TP), data and expert parallelism (DEP), and decode context parallelism (DCP).
• State Capture: The tool captures a complete snapshot of the workload, including CPU and GPU memory, which allows it to skip repetitive initialization steps like weight loading and torch.compile execution.
• Scalability: The system is capable of handling massive mixture-of-experts (MoE) models with checkpoint sizes reaching up to 1 TB across eight GPUs.
• Performance Benchmarks: In testing on NVIDIA B200 GPUs, the restore process achieved 8x to 36x speedups over traditional cold starts, with the GPU state restoration taking between 18 and 29 seconds.

Why it matters

Why it matters: This release provides a standardized way for platform engineers to manage AI inference scaling on Kubernetes. By offering ready-made examples for vLLM, SGLang, and NVIDIA TensorRT LLM, NVIDIA is facilitating faster recovery and more efficient resource utilization for large-scale AI deployments.