AMD ROCm team boosts GLM-5.2 AgentX serving 1.72x on MI355X via SGLang cache and kernel work

AMD's ROCm engineering team published a detailed optimization study showing substantial gains in serving the GLM-5.2 large language model with MXFP4 weights on Instinct MI355X GPUs. Using the open-source SGLang inference framework and the InferenceX AgentX benchmark, they raised peak per-GPU throughput from 10,625 to 18,327 tokens/s/GPU and improved peak P90 interactivity from 110.25 to 271.00 tokens/s/user. The work focused on two distinct performance regimes: below concurrency 12 where GPU-resident KV cache suffices, and above it where HiCache's hierarchical cache with host DRAM offload prevents repeated prefill. Multiple upstream pull requests to SGLang and ROCm enabled the improvements, including JIT copy paths for FP8 MLA cache rows, fused DeepSeek Sparse Attention kernels, and corrected host-pointer handling.

The AgentX benchmark specifically stresses serving infrastructure with coding-agent workloads: median requests contain roughly 142,000 input tokens and 444 output tokens, with 44% of sessions spawning subagents. This creates extreme KV cache pressure from shared prefixes and long contexts. AMD's published recipe uses tensor-parallel and expert-parallel configurations tuned to load, with EAGLE multi-token prediction for speculative decoding. The results were validated through controlled A/B testing and full one-hour sweeps rather than isolated microbenchmarks.

ROCm and SGLang serving optimizations for agentic workloads

• HiCache hierarchical KV cache: ROCm work enabled just-in-time copy paths for 576-byte FP8 multi-head latent attention rows and K-only host pools, allowing KV data to offload to host DRAM when GPU cache capacity is exceeded above concurrency 12, reducing repeated prefill.
• HIP graph and kernel improvements: PR #37118 enabled DeepSeek Sparse Attention head-gate helpers for prefill graph capture; PR #37124 fused DSA metadata kernels and eliminated repeated query concatenation and weight operations from the decode path.
• MXFP4 weight format with FP8 KV cache: The served GLM-5.2 model uses microscaling 4-bit floating-point weights, while the KV cache and EAGLE draft head operate in FP8 and BF16 respectively, creating mixed-precision serving complexity.
• Speculative decoding configuration: The published recipe employs five-step EAGLE multi-token prediction with six draft tokens to accelerate decode, with TP8/EP1 at lower concurrency and TP4/EP4 with HiCache enabled at higher load.
• Measurement methodology: Results derive from two deployed configurations measured on different dates (August 11 and October 6, 2026) using InferenceX commits d204ac0 and b3c3c3b96, with GPU allocation changing from 16 to four units; the polygon-area visualization comparing to B200/B300 references is not an official InferenceX metric.

Why it matters

This work demonstrates that AMD's ROCm platform can achieve competitive inference performance on emerging agentic AI workloads through upstream open-source collaboration rather than proprietary stacks, with all optimizations contributed to SGLang. For organizations deploying coding agents or long-context LLM services on AMD hardware, the published recipe and telemetry guide provide a reproducible starting point that reduces vendor lock-in. The focus on open-source serving frameworks and published benchmarks also gives the broader AI infrastructure community visibility into how hardware-specific optimizations translate to real-world throughput and latency under sustained load.