NVIDIA TensorRT-LLM 1.4.0rc0 adds Mooncake KV store, sparse cache offload, breaks Python KVCacheManagerV2 API

NVIDIA has published TensorRT-LLM v1.4.0rc0, a pre-release of its optimized inference library for large language models. The update adds experimental Mooncake store pool integration for distributed KV cache management, introduces sparse KV cache configuration with tier-aware locking and GPU metadata publication, and lays groundwork for DeepSeek V4 sparse attention offload. Several breaking API changes affect serving infrastructure: the Python backend of KVCacheManagerV2 is removed entirely, and LlmRequest fields migrate to Python-native structures. Model support expands to GLM-5.3-Flash, K-EXAONE-2, Qwen-Image-Layered with FP8/Cache-DiT, and MiniMax-M3 with NVFP4 KV caching. The release also enables tiled FP8 context FMHA for head dimension 256 on Blackwell SM120 GPUs and adds per-request speculative-decoding acceptance reporting. Known issues include hangs with MPI worker pool reuse on H100 TP2 configurations and illegal memory access crashes with DeepSeek MLA on Blackwell when CUDA graphs use BF16 BMM.

What changes in TensorRT-LLM 1.4.0rc0

• Mooncake KV store integration: Adds a Mooncake-based store pool, CLI tooling, and V2 scheduler preemption support, enabling distributed KV cache pooling across nodes for disaggregated serving architectures.
• Sparse KV cache infrastructure: Introduces sparse KV cache configuration, tier-aware locking, GPU metadata publication, and preparatory work for DeepSeek V4 attention offload, reducing memory pressure for long-context models.
• Breaking API removals: Eliminates the Python backend of KVCacheManagerV2 entirely and migrates LlmRequest fields to Python-native structures, requiring downstream serving code updates.
• Blackwell SM120 optimizations: Enables tiled FP8 context FMHA for head dimension 256 on SM120/SM121 GPUs (RTX PRO 6000 Blackwell), though DeepSeek MLA models on these GPUs may crash with BF16 BMM CUDA graph capture.
• Expanded model and quantization support: Adds GLM-5.3-Flash, K-EXAONE-2 PyTorch backend, Qwen-Image-Layered with FP8 and Cache-DiT, static FP8 Cosmos3 checkpoints without re-quantization, and MiniMax-M3 with NVFP4 KV cache and query-length buffer reuse.

Why it matters

For developers running self-hosted LLM inference on NVIDIA hardware, this release advances the state of disaggregated serving with Mooncake-based KV cache pooling and sparse offload capabilities that could reduce memory costs for long-context workloads. However, the breaking removal of KVCacheManagerV2's Python backend and LlmRequest field migration will require engineering effort to adapt existing serving stacks, while the known Blackwell and H100 stability issues suggest production deployments should await the final release or patches.