AMD details FP8 GEMM kernel optimization for RDNA 4 GPUs in ROCm

AMD's ROCm blog published the second part of a series on optimizing GEMM kernels for RDNA 4, focusing on porting from FP16 to FP8. The post explains the FP8 WMMA instruction changes on gfx1201 (Radeon RX 9070 XT), where E4M3 operands accumulate into FP32 rather than FP16. Because each FP8 element is one byte instead of two, a 128-bit load moves 16 values instead of 8, and fragment size per lane drops from 16 bytes to 8 bytes. These changes let kernels use larger tiles and deeper K slices within the same LDS budget. The authors test four configurations against hipBLASLt on transformer-derived GEMM shapes, with Config 1 being a direct type swap and Configs 2-4 exploring progressively larger tiles enabled by the space savings. All measurements use ROCm 10.0. The accompanying repository provides a complete implementation.

What changes in FP8 WMMA on RDNA 4

• Operand width: FP8 E4M3 elements are one byte each versus two bytes for FP16, so elems_per_b128 doubles from 8 to 16, propagating through all LDS size and copy-count calculations.
• Fragment layout: An FP8 fragment holds 8 values in 8 bytes per lane (two 32-bit registers with four FP8 values each), half the 16-byte FP16 fragment; the builtin accepts two packed int vectors rather than an FP8 vector type.
• Accumulator precision: FP8 WMMA accumulates into FP32 with no FP16 accumulator variant; output matrix C is float in column-major layout with leading dimension M, matching hipBLASLt's output format.
• LDS utilization: The smaller element size enables larger tile dimensions and deeper K slices that did not fit the LDS budget at FP16, with four configurations tested from a direct port up to a search-space-optimized tile.
• Current limitations: The kernel uses raw FP8 values without per-tensor or per-row scale factors, providing a foundation for production quantized inference rather than a complete solution.

Why it matters

This work matters for open-source AI inference on AMD hardware because FP8 is increasingly the default numerical format for quantized large language models, and having well-documented, reproducible kernel optimizations in ROCm reduces dependence on proprietary CUDA stacks. The explicit comparison against hipBLASLt and the published reference implementation give independent developers a baseline for building competitive open-source inference engines on consumer RDNA 4 GPUs.