# AMD ROCm madengine adds trace-based profiling ladder for DeepSeek-V4-Flash on MI350X

> **Key takeaway:** AMD's madengine v2.3.0 introduces a one-field-change profiling methodology that isolates training bottlenecks across PyTorch operators, GPU kernels, and communication gaps for DeepSeek-V4-Flash on Instinct MI350X.

**Published:** 2026-11-05T18:02:42.753384+00:00 
**Source:** AMD ROCm Official Blog 
**Category:** ai-local-edge 
**Canonical URL:** https://fosswire.org/news/amd-rocm-madengine-adds-trace-based-profiling-ladder-for-dee.html 

## Summary
AMD has published a detailed methodology for profiling large language model training workloads using madengine, a ROCm-based performance analysis tool, with a focus on DeepSeek-V4-Flash running in the Primus framework on eight AMD Instinct MI350X GPUs. The post demonstrates how to distinguish between framework operator overhead, GPU kernel execution, inter-GPU communication, and idle gaps using four complementary tracing tools: PyTorch profiler, rocm-trace-lite, rocprofv3, and TraceLens. Each tool is selected through a single field change in the madengine configuration, enabling reproducible, comparable runs with unmixed collectors.

The workload is an eight-layer BF16 training proxy that preserves the model's representative attention and mixture-of-experts patterns without requiring full pretraining resources. The tensor-parallel and pipeline-parallel sizes are both set to 1, with expert-parallel size of 8 across the eight-GPU node, placing 32 experts per GPU. The post provides specific file outputs, measurement scopes, and a progression from unprofiled baseline through increasingly targeted profiling runs, yielding concrete CSV artifacts for GPU timelines, operator summaries, GEMM rates, and collective communication analysis.

## madengine profiling architecture and lens selection
- **Single-field lens selection:** madengine uses one JSON field in `--additional-context` to select among profiler collectors, keeping traces unmixed and filenames consistent across runs for direct comparison.
- **Four-tool trace hierarchy:** PyTorch profiler captures operators and collectives; rocm-trace-lite intercepts HSA runtime for kernel dispatch timestamps; rocprofv3 records full-process kernel and memory-copy JSON traces; TraceLens post-processes all trace formats into unified CSV outputs.
- **DeepSeek-V4-Flash proxy configuration:** The eight-layer BF16 workload uses three sliding-window attention layers (compression ratio 0), three compressed-sparse layers (ratio 4), and two heavily compressed layers (ratio 128), each with 256-expert MoE and top-6 routing.
- **Fixed parallel topology:** Expert-parallel size of 8 with tensor-parallel and pipeline-parallel both at 1, microbatch size 1, global batch size 8, and sequence length 4096, meaning each GPU processes 4096 tokens per step.
- **Trace output artifacts:** Each rocprofv3 run generates approximately 150 MB of JSON per rank; TraceLens produces `gpu_timeline.csv`, `ops_summary*.csv`, `GEMM.csv`, and `nccl_summary_long.csv` with matched column names across all lens types.

## Why it matters
This methodology gives ROCm users a reproducible, documented path to optimize large-scale MoE training without the tooling fragmentation that typically plagues multi-GPU profiling. For AI infrastructure engineers running AMD Instinct clusters, the ability to isolate bottlenecks to specific operators, kernels, or communication patterns with controlled single-variable changes reduces optimization cycle time. The public availability of the `rocm/primus:v26.7` container image and the MAD repository also lowers the barrier to entry for organizations evaluating AMD as an alternative to NVIDIA's NSight Systems for LLM training workloads.