AMD ROCm madengine adds trace-based profiling ladder for DeepSeek-V4-Flash on MI350X

AMD has published a detailed methodology for profiling large language model training workloads using madengine, a ROCm-based performance analysis tool, with a focus on DeepSeek-V4-Flash running in the Primus framework on eight AMD Instinct MI350X GPUs. The post demonstrates how to distinguish between framework operator overhead, GPU kernel execution, inter-GPU communication, and idle gaps using four complementary tracing tools: PyTorch profiler, rocm-trace-lite, rocprofv3, and TraceLens. Each tool is selected through a single field change in the madengine configuration, enabling reproducible, comparable runs with unmixed collectors.

The workload is an eight-layer BF16 training proxy that preserves the model's representative attention and mixture-of-experts patterns without requiring full pretraining resources. The tensor-parallel and pipeline-parallel sizes are both set to 1, with expert-parallel size of 8 across the eight-GPU node, placing 32 experts per GPU. The post provides specific file outputs, measurement scopes, and a progression from unprofiled baseline through increasingly targeted profiling runs, yielding concrete CSV artifacts for GPU timelines, operator summaries, GEMM rates, and collective communication analysis.

madengine profiling architecture and lens selection

• Single-field lens selection: madengine uses one JSON field in --additional-context to select among profiler collectors, keeping traces unmixed and filenames consistent across runs for direct comparison.
• Four-tool trace hierarchy: PyTorch profiler captures operators and collectives; rocm-trace-lite intercepts HSA runtime for kernel dispatch timestamps; rocprofv3 records full-process kernel and memory-copy JSON traces; TraceLens post-processes all trace formats into unified CSV outputs.
• DeepSeek-V4-Flash proxy configuration: The eight-layer BF16 workload uses three sliding-window attention layers (compression ratio 0), three compressed-sparse layers (ratio 4), and two heavily compressed layers (ratio 128), each with 256-expert MoE and top-6 routing.
• Fixed parallel topology: Expert-parallel size of 8 with tensor-parallel and pipeline-parallel both at 1, microbatch size 1, global batch size 8, and sequence length 4096, meaning each GPU processes 4096 tokens per step.
• Trace output artifacts: Each rocprofv3 run generates approximately 150 MB of JSON per rank; TraceLens produces gpu_timeline.csv, ops_summary*.csv, GEMM.csv, and nccl_summary_long.csv with matched column names across all lens types.

Why it matters

This methodology gives ROCm users a reproducible, documented path to optimize large-scale MoE training without the tooling fragmentation that typically plagues multi-GPU profiling. For AI infrastructure engineers running AMD Instinct clusters, the ability to isolate bottlenecks to specific operators, kernels, or communication patterns with controlled single-variable changes reduces optimization cycle time. The public availability of the rocm/primus:v26.7 container image and the MAD repository also lowers the barrier to entry for organizations evaluating AMD as an alternative to NVIDIA's NSight Systems for LLM training workloads.