# FlashInfer on ROCm 10.1 adds automatic kernel routing for AMD Instinct GPUs

> **Key takeaway:** FlashInfer brings a unified API to AMD hardware, using automatic routing to switch between in-tree HIP kernels and optimized AITER libraries.

**Published:** 2026-11-20T16:03:30.441868+00:00 
**Source:** AMD ROCm Official Blog 
**Category:** ai-local-edge 
**Canonical URL:** https://fosswire.org/news/flashinfer-on-rocm-101-adds-automatic-kernel-routing-for-amd.html 

## Summary
AMD has announced the integration of FlashInfer 1 into the ROCm 10.1 ecosystem, providing an open-source library of attention and LLM serving kernels for AMD Instinct GPUs. This release implements a unified API that mirrors the upstream CUDA implementation, allowing developers to use the same plan() and run() models across different hardware backends.

The implementation features an automatic routing mechanism (backend="auto") that selects the most efficient kernel for a specific operation based on the GPU architecture. It supports CDNA3 (MI300X/MI325X) and CDNA4 (MI350X/MI355X) architectures, leveraging both in-tree HIP kernels and AMD's specialized AITER library to optimize performance for large language model workloads.

## Technical implementation and kernel routing
- **Automatic Backend Routing:** The `backend="auto"` setting evaluates the operation type, data types, KV layout, and head dimensions to choose between AITER-optimized kernels and in-tree HIP kernels based on architecture-specific rules in `arch_caps.py`.
- **Hardware-Specific Optimizations:** In-tree kernels are custom-tuned for CDNA3 and CDNA4; for example, CDNA4 kernels utilize the `ds_read_b64_tr_b16` instruction to perform transposing LDS reads in a single instruction, replacing multiple software-based register shuffles used on CDNA3.
- **Expanded Kernel Support:** This release introduces 8-bit floating-point (FP8) prefill, POD-Attention (overlapping prefill and decode), fused Mixture-of-Experts (MoE), and support for Multi-Head Latent Attention (MLA) via the AITER backend.
- **Integration Depth:** AITER is integrated at the C++ level to launch kernels directly, minimizing Python overhead, while the library supports `torch.compile` when the `FLASHINFER_USE_TORCH_CUSTOM_OPS` environment variable is enabled.

## Why it matters
By providing a CUDA-compatible API, FlashInfer reduces the friction for developers migrating LLM serving workloads from NVIDIA to AMD Instinct hardware. The automatic routing and support for advanced features like FP8 and MoE ensure that high-performance AI applications can run on ROCm without requiring significant code rewrites.