llama.cpp v0.9.0 adds OCP FP8 quantization and multi-node tensor parallelism

The llama.cpp project has released version 0.9.0, a major update that significantly expands its quantization capabilities and hardware acceleration. A primary technical addition is the support for the OCP FP8 E4M3 tensor type, which allows for more efficient low-precision inference. The update also introduces a new flash_attn_ext_rows operation to improve unified KV cache efficiency during multi-sequence inference tasks.

Beyond quantization, the release enhances multi-modal capabilities by adding Chatterbox TTS support and Nemotron 3.5 transcription for 40 languages. The underlying ggml library has been updated to v0.29.0, bringing NCCL-based multi-node tensor parallelism and performance optimizations across a wide array of backends, including CUDA, Metal, Vulkan, and Hexagon. New architecture support has also been added for models including Mistral Large 4 and Metis-1.6.

Technical breakdown of llama.cpp v0.9.0

• Quantization and Precision: Implementation of the OCP FP8 E4M3 tensor type with saturating mode and conversion helpers, alongside NVFP4 GGUF support for LFM3 models.
• Compute Backend Optimizations: ggml v0.29.0 introduces SVE kernels for CPU, DPAS flash attention for SYCL, fused MoE MMQ for CUDA, and split-K tuning for Vulkan.
• Distributed Inference: Addition of a multi-process communicator interface enabling multi-node tensor parallelism through NCCL.
• KV Cache and Attention: New flash_attn_ext_rows operator for efficient multi-sequence unified KV caches and the retention of Hadamard rotation constants on the GPU.
• Multi-modal Expansion: Integration of Chatterbox TTS, Nemotron 3.5 offline transcription with speaker attribution, and improved Qwen3-VL video prompt template support.

Why it matters

This release lowers the barrier for high-performance local AI deployment by providing standardized FP8 support and multi-node scaling. It benefits developers building edge AI applications through improved audio/video processing and offers system administrators more efficient ways to run large-scale models across distributed hardware.