<p>Large training jobs can fail even when every node looks healthy. A GPU with a subtle fault can pass its stress tests and still return wrong values. When a job stalls on an NCCL timeout, it keeps its GPUs, and Slurm still shows it as running. If a device plugin crashes on one node, the job fails with an error that never mentions the plugin. Teams usually find these problems only after the loss diverges or the job fails, and by then the compute is spent. Each of these failures lowers goodput, the share of your cluster’s time that goes to useful training. Today, we’re introducing workload recovery and infrastructure checks on Amazon SageMaker HyperPod that catch these failures sooner and raise the goodput of your training workloads.</p> <p>SageMaker HyperPod already handles nodes that report hardware faults. The health monitoring agent (HMA) watches each node for GPU errors, deep health checks test nodes during provisioning, and automatic node recovery reboots or replaces faulty hardware. These features look at one node’s hardware at a time. They can’t tell whether the GPUs train correctly, whether a job is still making progress, or whether the Kubernetes components a job depends on are working.</p> <p>In this post, we walk through the three new capabilities that catch these failures. For each one, we show how to set it up and what it reports when it finds a problem:</p> <ol type="1"> <li><code>MultiNodeTrainingValidation</code> is a new deep health check for HyperPod clusters orchestrated by Amazon Elastic Kubernetes Service (Amazon EKS). When HyperPod provisions an instance, or when you run the check on demand, it trains a small model on the instance and then across instances of the same type. The instance fails if the loss becomes NaN or infinite, or training doesn’t finish in time.</li> <li>The HyperPod Kubernetes Resource Monitoring Agent watches the Kubernetes layer of an EKS cluster for problems such as crash-looping device plugins, missing DaemonSets, and tainted nodes. When a job fails, it reports the pods and nodes involved as Kubernetes events and Amazon CloudWatch logs.</li> <li>Hung job detection finds Slurm jobs that have stopped making progress. An agent on each node watches the job’s output and GPU activity, or log patterns that you define. It writes an event to CloudWatch and can cancel the job so that Slurm requeues it.</li> </ol> <h2 id="why-node-health-alone-isnt-enough">Why node health alone isn’t enough</h2> <p>The existing HyperPod resiliency features check a single node at a time. The health monitoring agent (HMA) watches each node for GPU errors such as Xid and NVLink faults, Elastic Fabric Adapter (EFA) errors, and kernel errors. The InstanceStress deep health check runs DCGM level 4 diagnostics and tests each instance’s GPUs, GPU memory, NVLink, and EFA devices. The InstanceConnectivity check runs NCCL tests across GPU instances, or NCCOM tests on AWS Trainium. Together, these confirm that each node’s hardware works. They don’t show whether the GPUs compute correctly, whether a job is still making progress, or whether the cluster’s Kubernetes components are healthy. Three kinds of failure get past them:</p> <ul> <li><strong>Numerical faults –</strong> A GPU can pass every hardware test and still compute wrong results. The first sign is often NaN or infinite loss in a production run. Meta attributed 6 of the 419 unexpected interruptions in its 54-day Llama 3 405B pre-training run to silent data corruption (<a href="https://arxiv.org/abs/2407.21783" target="_blank" rel="noopener">The Llama 3 Herd of Models</a>).</li> <li><strong>Hung jobs –</strong> An NCCL timeout, a stalled data pipeline, or a deadlock can stop a job without ending it. The job keeps its GPUs and their memory, and Slurm reports it as running until someone checks the loss curve or the log.</li> <li><strong>Cluster-level failures –</strong> Some failures happen in Kubernetes, not in a node’s hardware. A device plugin crash-loops, a required DaemonSet never starts on a new node, or ranks on different nodes fall out of step in an NCCL collective. HMA checks each node on its own, so it doesn’t see these problems. When the job fails, the error often doesn’t say which pod or node caused it.</li> </ul> <h2 id="solution-overview">Solution overview</h2> <p>Each capability covers a different point in a training run. <code>MultiNodeTrainingValidation</code> runs when instances are provisioned, or on demand, and confirms that their GPUs train correctly. Hung job detection runs alongside each Slurm job and catches a job that has stopped making progress. The HyperPod Kubernetes Resource Monitoring Agent watches the cluster continuously and, when a job fails, reports the pods and nodes that were involved. The following table summarizes the three.</p> <table border="1px" width="100%" cellpadding="10px"> <tbody> <tr> <td><strong>Capability</strong></td> <td><strong>What it catches</strong></td> <td><strong>When it runs</strong></td> <td><strong>Available on</strong></td> </tr> <tr> <td><code>MultiNodeTrainingValidation</code> deep health check</td> <td>NaN or infinite loss, or training that doesn’t finish in time, on GPUs that pass hardware checks</td> <td>When instances are created or updated, or on demand</td> <td>EKS clusters, on supported EFA-enabled NVIDIA GPU instances</td> </tr> <tr> <td>HyperPod Kubernetes Resource Monitoring Agent</td> <td>Node condition changes, pod and DaemonSet failures, and failed training jobs, with the pods and nodes involved. NCCL communication state for jobs that opt in</td> <td>Continuously</td> <td>EKS clusters</td> </tr> <tr> <td>Hung job detection</td> <td>Jobs that stop making progress but don’t exit</td> <td>While each job runs</td> <td>Slurm clusters</td> </tr> </tbody> </table> <p>Hung job detection is new for Slurm clusters. On EKS clusters, HyperPod already detects hung jobs that run through the <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/sag
- For each one, we show how to set it up and what it reports when it finds a problem:</p> <ol type="1"> <li><code>MultiNodeTrainingValidation</code> is a new deep health check for HyperPod clusters orchestrated by Amazon Elastic Kubernetes Service (Amazon EKS).
- Three kinds of failure get past them:</p> <ul> <li><strong>Numerical faults –</strong> A GPU can pass every hardware test and still compute wrong results.
Nguyên văn nguồn Chi tiết nguồn