vLLM and SGLang merge weight daemons, engine snapshots to cut LLM inference startup time

The Cloud Native Computing Foundation has published an analysis of emerging techniques to reduce large language model inference startup latency, focusing on merged and in-development features in vLLM and SGLang. Both projects now incorporate persistent daemons that load and process model weights once, holding tensors in GPU memory for reuse across engine restarts via CUDA inter-process communication. vLLM has merged its weight-daemon implementation alongside a vllm preload command and experimental CRIU-based initialized-engine snapshots that capture Linux process and NVIDIA GPU state. SGLang has similarly merged a weight-cache daemon and published a Fast Engine Recovery roadmap, with an open RFC for initialized-engine snapshots captured before HTTP service binding.

These developments address a critical operational bottleneck in Kubernetes and cloud-native deployments: every new replica, restart, or model switch currently repeats work ranging from file loading and tensor construction through kernel compilation, autotuning, and CUDA Graph capture. The article categorizes optimization approaches by what state they preserve—from model files and compiled artifacts through processed weights, live engines, and full initialized execution state—showing how these layers can compose but remain independently developed across the OCI, Kubernetes, container runtime, and serving platform boundaries.

Weight daemons, sleep/wake, and snapshot architectures

• Persistent weight daemons: vLLM's merged weight daemon and SGLang's merged weight-cache daemon both load and process model weights, retain resulting tensors in GPU memory, and expose them to new engine processes through CUDA IPC; vLLM's zero-copy mode maps existing daemon allocations without data movement.
• Sleep/wake lifecycle: vLLM's sleep mode retains a live server while releasing much GPU memory, providing a different optimization path from cold-start weight reuse; the preload documentation notes restrictions on combining zero-copy preload with sleep-mode weight offloading.
• Initialized-engine snapshots: vLLM's merged experimental snapshot workflow uses CRIU for Linux process state and NVIDIA CUDA checkpoint support for GPU state, scoped to single-node single-GPU deployments with tensor parallelism of one; it releases weights and KV cache before capture and reloads them during restoration.
• SGLang snapshot RFC: The open proposal targets capture after initialization but before HTTP service binding, with initial scope limited to one NVIDIA GPU, tensor parallelism of one, and the same host and software environment; memory release/resume and generation pause/continue operations support external snapshot coordination.
• Complementary loading optimizations: SGLang documents pre-sharded state, fastsafetensors, and streaming loaders; llm-d documents model-file caches; vLLM's preload can prepare FlashInfer autotune results before engine start, addressing compilation and tuning latency separately from weight transfer.

Why it matters

For platform engineers running LLM inference on Kubernetes, these merged daemons and snapshot prototypes directly address autoscaling latency and resource efficiency: weight daemons eliminate redundant GPU memory transfers during scale-out, while sleep/wake and snapshot techniques reduce the time from pod scheduling to request serving. The divergence in approaches—vLLM's CRIU-based process snapshots versus SGLang's pre-bind capture, CUDA IPC weight sharing versus potential CPU backup policies—means operators will need to evaluate compatibility with their container runtimes, node-local storage, and multi-GPU tensor-parallel deployments as these features mature beyond single-GPU scopes.