vLLM spotlight: Berkeley-born inference engine now supports 200+ model architectures
vLLM, a high-throughput LLM inference engine originally from UC Berkeley's Sky Computing Lab, has grown into one of the most active open-source AI projects with over 2000 contributors.
Summary
vLLM is an open-source inference and serving engine for large language models that emphasizes throughput and memory efficiency through its PagedAttention memory management. The project supports NVIDIA, AMD, Intel, and Apple Silicon GPUs plus Google TPUs and other hardware through plugin architecture. It offers an OpenAI-compatible API server and integrates with Hugging Face models.
Architecture and hardware support
- PagedAttention: Efficient key-value cache memory management that reduces waste during LLM serving
- Continuous batching with chunked prefill, prefix caching, and speculative decoding (n-gram, EAGLE, DFlash)
- Quantization support: FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, TorchAO
- Parallelism modes: Tensor, pipeline, data, expert, and context parallelism for distributed inference
- Hardware coverage: NVIDIA, AMD, Intel GPUs; x86/ARM/PowerPC CPUs; plugins for Google TPU, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, MetaX GPU
- Model support: 200+ Hugging Face architectures including Llama, Qwen, Gemma, Mixtral, DeepSeek-V3, Mamba, LLaVA, embedding models
Why it matters
vLLM has become a standard deployment option for organizations running open-weight LLMs, offering a production-ready alternative to proprietary serving stacks with broad hardware and model coverage.
Original reference: github.com →