AMD ROCm blog details GLM-5.3-Flash inference on MI300X with TP8 and DP8+EP8 parallelism

AMD's ROCm blog published a detailed analysis of serving GLM-5.3-Flash, a 320-billion-parameter sparse mixture-of-experts language model, on a single node of eight AMD Instinct MI300X accelerators. The post examines how two parallelism strategies—TP8 (tensor parallelism across eight GPUs) and DP8+EP8 (data parallelism plus expert parallelism)—perform under long-context, high-concurrency inference workloads where the KV cache becomes the dominant memory consumer. The model's FP8 weights occupy approximately 306 GB, leaving most of the node's 1,536 GB of HBM for the KV cache that stores attention keys and values for active requests.

The analysis reveals a fundamental latency-throughput tradeoff: TP8 delivers lower per-request latency by distributing each layer's compute across all GPUs, but replicates the KV cache on every GPU, severely limiting total cached tokens. DP8+EP8 keeps only one copy of each request's cache, enabling far more concurrent long sessions at the cost of slower individual requests. The blog also evaluates BF16 versus FP8 KV-cache precision, finding that FP8 nearly doubles cache capacity with less than 2% throughput change until the BF16 pool saturates. Measurements cover three request shapes: 1K in/1K out, 8K in/1K out, and 96K in/32K out.

GLM-5.3-Flash architecture and parallelism tradeoffs

• Sparse attention architecture: Only 11 of 45 layers (every fourth layer starting at layer 3) use DeepSeek sparse attention with Multi-head Latent Attention and store a growing KV cache; the other 34 layers use Kimi Delta Attention with fixed recurrent state, keeping per-token cache costs roughly one-quarter of a full-cache design.
• KV cache structure: Each cached token occupies a 512-element latent vector per layer with no RoPE half (qk_rope_head_dim is 0), plus a small 128-dimensional indexer key cache pooled by four; this compact representation enables millions of cached tokens on the eight-GPU node after weight residency.
• TP8 limitations: Tensor parallelism shards model weights and compute across all eight GPUs but replicates the full KV cache on each GPU, yielding the lowest single-request latency but capping total cached tokens to one GPU's worth—reported as 9,841,634 tokens in BF16 at boot.
• DP8+EP8 advantages: Data parallelism with expert parallelism runs eight independent ranks serving separate requests while sharding routed experts, storing each request's cache exactly once and dramatically expanding the node's total cache capacity for concurrent long-context sessions.
• FP8 KV-cache precision: Storing cached values in one byte instead of two nearly doubles token capacity with measured throughput change under 2% while BF16 capacity suffices, and provides throughput gains once BF16 memory is exhausted and requests would otherwise queue.

Why it matters

This ROCm publication demonstrates competitive AMD GPU support for state-of-the-art Chinese open-weight models and provides reproducible tuning guidance for operators deploying large MoE models without NVIDIA hardware lock-in. The detailed memory and parallelism analysis helps vLLM and ROCm users optimize serving infrastructure for long-context applications such as document analysis and coding assistants, while the FP8 KV-cache findings offer immediate operational savings for memory-constrained deployments.