FlashInfer on ROCm 10.1 adds automatic kernel routing for AMD Instinct GPUs
FlashInfer brings a unified API to AMD hardware, using automatic routing to switch between in-tree HIP kernels and optimized AITER libraries.
Summary
AMD has announced the integration of FlashInfer 1 into the ROCm 10.1 ecosystem, providing an open-source library of attention and LLM serving kernels for AMD Instinct GPUs. This release implements a unified API that mirrors the upstream CUDA implementation, allowing developers to use the same plan() and run() models across different hardware backends.
The implementation features an automatic routing mechanism (backend="auto") that selects the most efficient kernel for a specific operation based on the GPU architecture. It supports CDNA3 (MI300X/MI325X) and CDNA4 (MI350X/MI355X) architectures, leveraging both in-tree HIP kernels and AMD's specialized AITER library to optimize performance for large language model workloads.
Technical implementation and kernel routing
- Automatic Backend Routing: The
backend="auto"setting evaluates the operation type, data types, KV layout, and head dimensions to choose between AITER-optimized kernels and in-tree HIP kernels based on architecture-specific rules inarch_caps.py. - Hardware-Specific Optimizations: In-tree kernels are custom-tuned for CDNA3 and CDNA4; for example, CDNA4 kernels utilize the
ds_read_b64_tr_b16instruction to perform transposing LDS reads in a single instruction, replacing multiple software-based register shuffles used on CDNA3. - Expanded Kernel Support: This release introduces 8-bit floating-point (FP8) prefill, POD-Attention (overlapping prefill and decode), fused Mixture-of-Experts (MoE), and support for Multi-Head Latent Attention (MLA) via the AITER backend.
- Integration Depth: AITER is integrated at the C++ level to launch kernels directly, minimizing Python overhead, while the library supports
torch.compilewhen theFLASHINFER_USE_TORCH_CUSTOM_OPSenvironment variable is enabled.
Why it matters
By providing a CUDA-compatible API, FlashInfer reduces the friction for developers migrating LLM serving workloads from NVIDIA to AMD Instinct hardware. The automatic routing and support for advanced features like FP8 and MoE ensure that high-performance AI applications can run on ROCm without requiring significant code rewrites.
What to do
- Use the backend="auto" setting to allow FlashInfer to automatically select between AITER and in-tree HIP kernels based on your GPU architecture.
- Enable the FLASHINFER_USE_TORCH_CUSTOM_OPS environment variable to support torch.compile.
Original reference: rocm.blogs.amd.com →