AMD open-sources megakernel LLM inference paths for MI355X, up to 3.3x faster decode

AMD has published open-source optimized inference paths for three large language models running on its Instinct MI355X GPUs, targeting the latency-sensitive interactive use cases that now dominate AI deployment. The implementations integrate with existing open-source serving engines ATOM, SGLang and vLLM rather than replacing them. For Qwen3.8-27B, the optimized path reaches approximately 1,100 tokens per second, doubling ATOM's regular path performance. Qwen3.8-Flash-Next achieves roughly 750 tokens per second, a 3.3x improvement over ATOM's standard implementation. GPT-OSS-120B, offered as a preview in vLLM, hits about 1,470 tokens per second, 1.9x faster than stock vLLM in AMD's testing. The optimizations were developed through hundreds of experiments by AMD engineers assisted by AI coding agents. The blog post details reproduction steps for each configuration.

Four techniques behind the MI355X speedups

• Megakernels reduce launch overhead: Instead of launching separate kernels for each operator in each layer, a persistent kernel stays resident on each compute unit and workgroups hand data through memory. Qwen3.8-27B and GPT-OSS-120B run the entire forward pass as one kernel; Qwen3.8-Flash-Next uses one kernel per decoder layer with only ~1.7 µs grid-wide synchronization between layers.
• Producer-consumer handoffs with fine-grained synchronization: Stage boundaries pass a value and sequence tag in a single 64-bit store, letting consumers poll exactly the data they need. Per-chiplet release words keep most polling in local cache, and idle compute units preload upcoming weights before the next stage's input arrives.
• Speculative decoding multiplies effective throughput: All three optimized paths use speculative decoding with draft models. Qwen3.8-27B uses the same drafter model in both regular and optimized paths for the 2.0x comparison; Qwen3.8-Flash-Next uses an MTP drafter with 3 drafts for its 3.3x speedup; GPT-OSS-120B uses EAGLE3 in the optimized path versus DFlash in stock vLLM.
• MI355X dual-die data placement (DPX+NPS2): Running the GPU as two logical devices, one per die, keeps each die's data in local memory. GPT-OSS-120B is the first model here to exploit this mode, gaining 7.5-8% from data placement alone and 16-20% when combined with speculative decoding.
• Sparse attention optimization in Qwen3.8-Flash-Next: The model's sparse attention scores every context block and selects the top 512; parallelizing this selection and merging winners reduced a 250K-token sparse-attention layer from 262 µs to 126 µs, with further tuning bringing a single token at 250K context down to 3.0 ms versus 10.2 ms on the regular path.

Why it matters

These open-source paths give Linux-based AI infrastructure operators concrete, reproducible optimizations for AMD's newest accelerator without waiting for upstream engine maintainers to catch up. By publishing integration patches for ATOM, SGLang and vLLM alongside detailed methodology, AMD reduces the fragmentation that typically plagues non-CUDA inference stacks and provides a template for how hardware vendors can contribute latency-focused optimizations directly to open-source serving ecosystems.