AMD ROCm adds GPU-initiated SDMA to RCCL and rocSHMEM for MI350X

AMD has extended its open-source ROCm communication stack with GPU-initiated SDMA (System DMA) copy engine support in both RCCL and rocSHMEM, enabling GPU kernels to directly initiate data transfers without consuming compute units for the actual movement work. Previously, fused compute-communication kernels on AMD Instinct hardware either used CUs executing load/store instructions or relied on CPU-initiated SDMA transfers; the new approach uses a single CU to issue commands to dedicated SDMA engines that handle the copy. Benchmarks on eight AMD Instinct MI350X GPUs demonstrate that this GIN-SDMA path matches or exceeds the bandwidth of RCCL's default 64-CU AlltoAll and AllGather implementations for large messages, while rocSHMEM putnbi tests show SDMA becoming the highest-bandwidth option at 512 KiB and above.

The capability is particularly targeted at inference workloads where communication latency sits on the critical path and traditional compute-communication overlap techniques fail without fine-grained kernel-level orchestration. Both libraries expose device-side APIs that allow GPU kernels to fuse computation with communication initiation, with RCCL adding this as a new backend option and rocSHMEM having been architected for GPU-initiated operation from its inception. The implementation requires symmetric memory registration for the SDMA paths, unlike the default RCCL load/store approach.

How GPU-initiated SDMA changes data movement on AMD Instinct

• SDMA engine utilization vs. CU load/store: The new path uses dedicated SDMA hardware engines for actual data movement rather than consuming CUs to execute copying instructions, with only one CU needed to issue commands and poll completion.
• RCCL backend selection logic: RCCL's GIN-SDMA backend activates for AlltoAll messages above 32 MiB and AllGather messages at 512 KiB or larger, falling back to vector load-store operations for smaller transfers where setup overhead dominates.
• rocSHMEM wavefront efficiency: rocSHMEM with SDMA enabled achieves higher bandwidth using one CU and one wavefront than the SDMA-disabled configuration using one CU with 16 wavefronts, demonstrating that additional thread parallelism cannot compensate for lacking dedicated copy engines.
• Symmetric memory registration requirement: Both Host-SDMA and GIN-SDMA paths in RCCL require symmetric memory registration, trading setup complexity against runtime CU efficiency compared to the unregistered default implementation.
• Scale-up domain saturation: AMD Instinct MI350X GPUs carry sufficient SDMA engine count to fully saturate intra-node (scale-up) interconnect bandwidth, making the single-CU command overhead negligible for large-message throughput.

Why it matters

For ROCm-based AI inference deployments and HPC applications, this reduces the fundamental tension between communication bandwidth and compute availability that has plagued fused kernels on GPU architectures. The open-source implementation in RCCL and rocSHMEM gives AMD Instinct users a path to higher effective throughput without proprietary networking stacks, though the symmetric memory registration requirement will need framework-level support to become transparent to PyTorch and TensorFlow users.