NVIDIA TensorRT-LLM v1.4.0rc3 adds Blackwell support and native sparse MLA attention

Brief written by the FOSSWire AI editor from NVIDIA TensorRT-LLM How we write

November 6, 2026 1 min read

Key Takeaway

The release introduces significant optimizations for Blackwell architecture, native sparse attention for DeepSeek-V4, and expanded multimodal support via OpenEngine.

Summary

NVIDIA has released the v1.4.0rc3 pre-release of TensorRT-LLM, a high-performance library for accelerating Large Language Model (LLM) inference. This update focuses heavily on hardware-specific optimizations for the Blackwell architecture, including support for GLM-5.2 and MiniMax-M3 with NVFP4 low-latency decoding.

Beyond hardware acceleration, the release expands the library's capabilities in multimodal processing by adding image, video, and audio inputs to the OpenEngine gRPC API. It also introduces foundational improvements to the KV cache management system and native sparse Multi-Head Latent Attention (MLA) for DeepSeek-V4 generation. These changes aim to improve throughput and reduce latency for complex, large-scale Mixture-of-Experts (MoE) models.

Technical advancements in v1.4.0rc3

Architecture Optimization: Adds support for Blackwell (SM103/SM120) targets, including NVFP4 modeling for GLM-5.2 and accelerated serving for Qwen3.5 and GLM-5.3-Flash.
Attention Mechanisms: Implements native sparse MLA attention for DeepSeek-V4 and introduces a unified KV cache manager with sparse attention arenas and host-backed pools.
Multimodal & API Expansion: Enhances the OpenEngine gRPC interface to accept audio, video, and image inputs, while exposing Prometheus metrics and standard health checks.
Memory & Throughput: Introduces native FP8 MoE LoRA weight support and optimizes speculative decoding through vocabulary-parallel selection and DSpark verification.
Deployment Fixes: Resolves a critical dependency issue by removing direct Git requirements, enabling the publication of PyPI wheels for easier distribution.

Why it matters

This release provides developers with deeper integration for the latest NVIDIA Blackwell hardware, essential for scaling high-performance AI workloads. The improvements to KV cache management and sparse attention directly benefit the deployment of massive MoE models like DeepSeek. Furthermore, the move toward PyPI wheel compatibility simplifies the installation process for AI engineers working in Python-centric environments.

What to do

Users of DSA models like GLM-5.2 should be aware that they may produce incorrect output when using an NVFP4 KV cache.
DeepSeek-V4-Pro users should monitor for GPU memory exhaustion during initialization if MPI workers are reused after large MoE workloads.
Users of Kimi K3 DSpark with MegaMoE CuTe DSL TEP and CUDA graphs should watch for hangs during GB300 warmup.
Tensor-parallel serving users with RPC orchestration should watch for hangs during model initialization when reusing an MPI worker pool across threads.

Original reference: github.com →