# NVIDIA releases NVIDIA Recovery open-source service for automated AI training workloads

> **Key takeaway:** NVIDIA Recovery automates the detection, attribution, and restart of distributed AI training jobs to minimize downtime caused by hardware or software failures.

**Published:** 2026-11-02T22:43:09+00:00 
**Source:** NVIDIA Technical & Open Source Blog 
**Category:** bigtech-foss 
**Canonical URL:** https://fosswire.org/news/nvidia-releases-nvidia-recovery-open-source-service-for-auto.html 

## Summary
NVIDIA has announced the release of NVIDIA Recovery, an open-source workload-level service designed to maintain continuous distributed AI training. The service addresses the growing problem of training disruptions caused by GPU node failures, NCCL timeouts, and straggler nodes in large-scale clusters. By monitoring application logs, GPU metrics, and scheduler signals, the tool automates the recovery loop to sustain training goodput.

NVIDIA Recovery functions as a coordination layer within the DSX OS stack, integrating with existing infrastructure remediation and framework-level libraries. It is designed to work with Slurm via an Advisor Controller deployable through Helm charts, supporting both in-cluster and SSH modes. The service includes a dedicated Recovery Console for real-time monitoring and an audit trail for failure attribution and action history.

## Technical architecture and recovery loop
- **Four-stage recovery loop:** The service operates through a continuous cycle of anomaly detection, failure attribution, node isolation, and workload restart or stoppage based on user-defined policies.
- **Slurm integration:** Deployment is facilitated via a reference Advisor Controller using Helm charts, allowing the service to remain independent of internal scheduler logic while managing job restarts and placement.
- **Observability and monitoring:** New deployments default to a 'monitor mode' that provides recovery recommendations without executing scheduler actions, allowing operators to validate signal quality before enabling autonomy.
- **Multi-layer integration:** The service coordinates with NVSentinel for GPU health monitoring and NVRx for in-job hang detection, bridging the gap between infrastructure-level fixes and job-level restarts.
- **Performance metrics:** In production testing across nearly 200 daily jobs, the system achieved a mean detection latency of 69 seconds and a mean recovery-action latency of 94 seconds.

## Why it matters
By open-sourcing the NVIDIA Recovery repository, NVIDIA provides cluster administrators and AI engineers with a tool to reduce wasted GPU-hours and manual intervention during large-scale training runs. This modular approach allows users to integrate automated recovery into existing Slurm or Kubernetes-based workflows, improving the reliability of distributed AI workloads.