NVIDIA releases NVIDIA Attribution open-source tool for distributed AI failure diagnosis

NVIDIA has open-sourced NVIDIA Attribution, a workload-centric diagnostic tool designed to solve the problem of cascading errors in distributed AI training. In large-scale GPU clusters, a single hardware or fabric event often triggers a massive wave of downstream symptoms, such as NCCL errors across multiple ranks, which can obscure the actual point of failure. This tool moves beyond isolated node reporting to provide a structured diagnosis of the entire workload.

The release consists of two primary components: the Attribution Service, which provides a REST API for the core engine, and the Attribution Blueprint, a Python-based orchestrator. By integrating with existing cluster schedulers and telemetry backends, the system can distinguish between a primary root cause—such as an NVLink signal integrity issue—and the resulting communication errors that follow. The software is already utilized within NVIDIA's own AI infrastructure to manage millions of weekly workloads.

Architecture and diagnostic workflow

• Core Components: The system utilizes the Attribution Service for the central engine via REST API and the Attribution Blueprint (Python) to orchestrate connections between schedulers and telemetry.
• Integration Connectors: The Blueprint includes execution-context connectors for Slurm and Kueue, alongside log and metric connectors for Loki, Kubernetes pod logs, local files, and PromQL metrics.
• Four-Stage Pipeline: The diagnostic process involves scoping the workload via scheduler identifiers, collecting observations from GPU drivers and Fabric Manager, extracting events through deterministic rules, and returning a structured result.
• Root Cause Correlation: The engine is designed to separate terminal symptoms, such as NCCL failures, from the actual trigger, such as a specific GPU XID error, to provide a clear evidence trail.

Why it matters

This release provides system administrators and AI engineers with a standardized way to debug complex fabric and GPU failures in large-scale clusters. By supporting open standards like PromQL and schedulers like Slurm and Kubernetes, it allows for more automated and reproducible infrastructure recovery in heterogeneous AI environments.