Local AI & Edge

FlashInfer on ROCm 10.1 adds automatic kernel routing for AMD Instinct GPUs

Key Takeaway

FlashInfer brings a unified API to AMD hardware, using automatic routing to switch between in-tree HIP kernels and optimized AITER libraries.

Summary

AMD has announced the integration of FlashInfer 1 into the ROCm 10.1 ecosystem, providing an open-source library of attention and LLM serving kernels for AMD Instinct GPUs. This release implements a unified API that mirrors the upstream CUDA implementation, allowing developers to use the same plan() and run() models across different hardware backends.

The implementation features an automatic routing mechanism (backend="auto") that selects the most efficient kernel for a specific operation based on the GPU architecture. It supports CDNA3 (MI300X/MI325X) and CDNA4 (MI350X/MI355X) architectures, leveraging both in-tree HIP kernels and AMD's specialized AITER library to optimize performance for large language model workloads.

Technical implementation and kernel routing

  • Automatic Backend Routing: The backend="auto" setting evaluates the operation type, data types, KV layout, and head dimensions to choose between AITER-optimized kernels and in-tree HIP kernels based on architecture-specific rules in arch_caps.py.
  • Hardware-Specific Optimizations: In-tree kernels are custom-tuned for CDNA3 and CDNA4; for example, CDNA4 kernels utilize the ds_read_b64_tr_b16 instruction to perform transposing LDS reads in a single instruction, replacing multiple software-based register shuffles used on CDNA3.
  • Expanded Kernel Support: This release introduces 8-bit floating-point (FP8) prefill, POD-Attention (overlapping prefill and decode), fused Mixture-of-Experts (MoE), and support for Multi-Head Latent Attention (MLA) via the AITER backend.
  • Integration Depth: AITER is integrated at the C++ level to launch kernels directly, minimizing Python overhead, while the library supports torch.compile when the FLASHINFER_USE_TORCH_CUSTOM_OPS environment variable is enabled.

Why it matters

By providing a CUDA-compatible API, FlashInfer reduces the friction for developers migrating LLM serving workloads from NVIDIA to AMD Instinct hardware. The automatic routing and support for advanced features like FP8 and MoE ensure that high-performance AI applications can run on ROCm without requiring significant code rewrites.

What to do

  • Use the backend="auto" setting to allow FlashInfer to automatically select between AITER and in-tree HIP kernels based on your GPU architecture.
  • Enable the FLASHINFER_USE_TORCH_CUSTOM_OPS environment variable to support torch.compile.

Original reference: rocm.blogs.amd.com →