NVIDIA TensorRT-LLM 1.4.0rc2 adds Rubin GPU support, FP4/FP8 inference, and breaking API changes

NVIDIA has released TensorRT-LLM v1.4.0rc2, a pre-release update to its open-source inference engine for large language models, with Python wheels published as 1.4.0rc2.post1 to resolve PyPI publication blockers. The release significantly expands hardware and model coverage, adding NVFP4 low-latency decode support for the upcoming Rubin architecture (SM103) across GLM-5.2, MiniMax-M3, GPT-OSS-120B, and DeepSeek-R1-0528 models, alongside FP8 and static NVFP4 paths for Cosmos3 and FLUX.1-dev visual generation workloads.

The release contains four documented breaking API changes: LlmRequest fields migrate from C++ to Python, KV cache manager V1 support is removed from transceiver V2, obsolete executor backend compatibility paths are deleted, and zero-copy mode is enabled for Mooncake Store. Additional features include native KV cache retention and cross-turn block reuse, FP4 MLA in disaggregated serving, MXFP8 refitting with CUDA graphs, and optimizations for DeepSeek-V4, Kimi K3, and Qwen3 vision-language encoders. A known issue remains with tensor-parallel RPC orchestration hanging during model initialization when reusing MPI worker pools across threads.

Architecture and inference changes in TensorRT-LLM 1.4.0rc2

• Rubin/SM103 architecture support: Adds NVFP4 low-latency decode defaults for GLM-5.2, GPT-OSS-120B, and DeepSeek-R1-0528, plus static NVFP4 with per-tensor FP8 attention for FLUX.1-dev, positioning the codebase for NVIDIA's next-generation datacenter GPUs.
• KV cache system overhaul: Native retention and cross-turn generation-block reuse are added to KV cache V2, with sliding-window prefix commits at fork points and first-N/last-N mixed-precision caching; FP4 MLA is now supported in disaggregated serving deployments.
• Breaking API migrations: LlmRequest fields move from C++ to Python structures, KV cache manager V1 is dropped from transceiver V2, obsolete executor backend paths are removed, and Mooncake Store gains zero-copy mode—changes that will require downstream integration updates.
• Model-specific optimizations: Fused RMSNorm/RoPE kernels for DeepSeek-V4 and DSpark on Rubin, shared CUDA graph setup for Qwen3-family vision-language encoders, prefill-chunk scheduling for multimodal encoders, and asynchronous Ulysses execution for MiniMax-H3.
• Precision and kernel improvements: MXFP8 refitting with piecewise CUDA graphs and Ray collectives, BF16 tensor-core router GEMMs for MiniMax-M3, SwiGLU fusion into BF16 gate/up GEMM epilogues on SM100, and default DSA paged-MQA indexing migrated to CuTe DSL.

Why it matters

For AI infrastructure engineers and cloud providers running local LLM inference, this release advances NVIDIA's open-source inference stack toward Rubin-era hardware while deepening the vendor-specific CUDA/NVFP4 dependency chain. The breaking API changes signal maturation of the KV cache V2 and Python-native serving paths, but will force synchronization between TensorRT-LLM deployments and downstream frameworks like vLLM or custom serving layers. The expanded Qwen3, GLM, and MiniMax model support reflects NVIDIA's continued prioritization of Chinese-origin model families in its optimization targets, relevant for global distribution strategies and competitive positioning against AMD ROCm and Intel OpenVINO ecosystems.