NVIDIA Triton Inference Server 2.74.0 updates vLLM backend and backend build requirements

NVIDIA has released version 2.74.0 of the Triton Inference Server, which corresponds to NGC container 26.10. This release provides an optimized inferencing solution for both CPU and GPU architectures, accessible via HTTP or gRPC endpoints or as a shared library for edge deployments.

The update includes significant changes to backend management, specifically requiring a TRITON_BUILD_CONTAINER for ONNX Runtime, OpenVINO, Python, and TensorRT-LLM builds. It also enhances the vLLM backend by adding support for version 0.30 or higher and implementing prompt rendering prior to generation and encoding.

Technical changes in Triton 2.74.0

• Backend Build Workflow: ONNX Runtime, OpenVINO, Python, and TensorRT-LLM backends now mandate the use of TRITON_BUILD_CONTAINER instead of synthesizing a py3-min image.
• vLLM Integration: The vLLM backend now supports versions >= 0.30 and includes a new process where prompts are rendered before the generation and encoding stages.
• Memory and Security: The Python backend now applies restrictive permissions to shared memory and implements a fallback to host memory for GPU tensor allocation.
• Request Management: Request cancellation checks have been implemented prior to backend dispatch and before batch assembly within the TensorRT backend.
• Infrastructure Updates: The server is now built upon and distributed using the cuda-dl-base devel image, and S3 addressing styles are now user-configurable.

Why it matters

This release is critical for AI engineers deploying large language models, as the updated vLLM support and prompt rendering improve inference workflows. However, developers must adapt to the new containerized build requirements and address potential security considerations regarding Ray and vLLM vulnerabilities in their network architectures.