# AMD ROCm blog publishes PyTorch GPU profiling guide with TinyOpenFold training example

> **Key takeaway:** AMD's seventh profiling guide installment applies rocprofv3, rocprof-compute, and rocprof-sys to PyTorch training workloads, demonstrating how to separate compiler warm-up from steady-state measurement and evaluate fusion, torch.compile, and custom Triton optimizations on ROCm.

**Published:** 2026-11-04T18:04:14.061527+00:00 
**Source:** AMD ROCm Official Blog 
**Category:** ai-local-edge 
**Canonical URL:** https://fosswire.org/news/amd-rocm-blog-publishes-pytorch-gpu-profiling-guide-with-tin.html 

## Summary
AMD has released the seventh entry in its ROCm GPU profiling guide series, shifting focus from HIP and Fortran applications to PyTorch training workloads on AMD Instinct hardware. The guide applies a systematic four-stage profiling loop—baseline timing, kernel hotspot analysis, hardware counter rooflines, and whole-system timelines—to TinyOpenFold, a scaled-down trainable reproduction of AlphaFold 2's Evoformer architecture. Unlike HPC applications compiled ahead of execution, PyTorch workloads compile during execution via torch.compile and Triton JIT compilation, requiring practitioners to separate warm-up iterations from measurement to avoid profiling the compiler rather than the model.

The guide demonstrates ROCTx annotation via torch.cuda.nvtx (which works on ROCm builds despite the CUDA namespace), iteration-multiplexed counter collection with rocprof-compute, and comparative evaluation of three optimization strategies: PyTorch-level operator fusion, torch.compile graph compilation, and custom Triton kernels with autograd integration. All code and documentation are available in AMD's HPCTrainingExamples repository under unspecified license terms, validated on ROCm 7.13 with PyTorch 2.11 and Triton 3.6 on MI300X accelerators.

## Profiling workflow and optimization evaluation
- **ROCTx annotation bridge:** The guide uses torch.cuda.nvtx for timeline markers on ROCm builds, leveraging PyTorch's device-agnostic API where ROCTx mirrors NVTX range calls, enabling portable profiling annotations across CUDA and ROCm runtimes without code changes.
- **Warm-up separation requirement:** torch.compile traces and generates kernels on the first iteration, while Triton JIT compiles and auto-tunes per tensor shape; the guide emphasizes isolating these one-time costs from steady-state steps to obtain valid performance measurements.
- **Iteration-multiplexed counter collection:** rocprof-compute gathers the full hardware counter set in a single training pass by multiplexing across iterations, avoiding the need for multiple runs to capture different counter groups.
- **Three-version comparison methodology:** TinyOpenFold provides a clean PyTorch baseline (version 1), PyTorch-native fusion with fewer dispatches and lower peak memory (version 2), and custom Triton kernels with fused LayerNorm and scaled dot-product attention wrapped in autograd functions (version 3), enabling controlled comparison of optimization strategies.
- **Triangle operation identification:** The profiler separates math-bound O(N_res³) triangle multiplication and self-attention stages from launch-bound overhead of small elementwise kernels, directing optimization effort toward the dominant compute patterns in the pair tensor updates.

## Why it matters
This guide reduces the barrier for PyTorch developers to optimize training workloads on AMD Instinct GPUs using the same profiling vocabulary already established for NVIDIA tools, lowering friction in multi-vendor AI infrastructure. By documenting how torch.compile and Triton behave on ROCm—including the critical warm-up separation step—it helps the open-source AI stack mature beyond CUDA-centric assumptions and supports genuine hardware portability for training frameworks.