NVIDIA releases NVIDIA Recovery open-source service for automated AI training workloads

NVIDIA has announced the release of NVIDIA Recovery, an open-source workload-level service designed to maintain continuous distributed AI training. The service addresses the growing problem of training disruptions caused by GPU node failures, NCCL timeouts, and straggler nodes in large-scale clusters. By monitoring application logs, GPU metrics, and scheduler signals, the tool automates the recovery loop to sustain training goodput.

NVIDIA Recovery functions as a coordination layer within the DSX OS stack, integrating with existing infrastructure remediation and framework-level libraries. It is designed to work with Slurm via an Advisor Controller deployable through Helm charts, supporting both in-cluster and SSH modes. The service includes a dedicated Recovery Console for real-time monitoring and an audit trail for failure attribution and action history.

Technical architecture and recovery loop

• Four-stage recovery loop: The service operates through a continuous cycle of anomaly detection, failure attribution, node isolation, and workload restart or stoppage based on user-defined policies.
• Slurm integration: Deployment is facilitated via a reference Advisor Controller using Helm charts, allowing the service to remain independent of internal scheduler logic while managing job restarts and placement.
• Observability and monitoring: New deployments default to a 'monitor mode' that provides recovery recommendations without executing scheduler actions, allowing operators to validate signal quality before enabling autonomy.
• Multi-layer integration: The service coordinates with NVSentinel for GPU health monitoring and NVRx for in-job hang detection, bridging the gap between infrastructure-level fixes and job-level restarts.
• Performance metrics: In production testing across nearly 200 daily jobs, the system achieved a mean detection latency of 69 seconds and a mean recovery-action latency of 94 seconds.

Why it matters

By open-sourcing the NVIDIA Recovery repository, NVIDIA provides cluster administrators and AI engineers with a tool to reduce wasted GPU-hours and manual intervention during large-scale training runs. This modular approach allows users to integrate automated recovery into existing Slurm or Kubernetes-based workflows, improving the reliability of distributed AI workloads.