1 tiếng nói · chưa gắn nhãn trưởng lab · 1 nền tảng · chưa có số tương tác · Nguồn crawl
M
Mijanur Palash 1 bài viết · AWS Machine Learning Blog Nguyên văn nguồn Chi tiết nguồn
<p>Large-scale model training is one of the most expensive workloads you can run on GPUs, and a single unhealthy accelerator can stall an entire distributed job. When training throughput drops, the hard part is rarely noticing that something is wrong. It is figuring out why and where. Teams often spend hours or days sifting through logs and disconnected dashboards to decide whether a slowdown came from a throttled GPU, a saturated network link, or a stalled data pipeline on the host.</p> <p>Today, we are announcing enhanced observability in Amazon SageMaker HyperPod, a new capability that helps you diagnose GPU training issues faster. It brings together diagnostic and profiling capabilities for training workloads that run on both Amazon Elastic Kubernetes Service (Amazon EKS) and Slurm orchestrators:</p> <p><strong>Diagnostic summary</strong> uses Amazon CloudWatch’s DetectGroupOutliers API to compare each GPU in a training job on utilization, memory, temperature, power, and SM clock, and flags GPUs that deviate from their peers. Static diagnostic rules surface common failure patterns such as GPU hardware faults (XID errors, ECC memory errors), EFA throughput drops, thermal throttling, and overheating, so you catch degradation before it affects training.</p> <h2 id="why-you-need-it">Why you need it</h2> <p>Distributed training is expensive, and small inefficiencies compound into significant wasted GPU hours. Until now, getting visibility into HyperPod infrastructure meant stitching together separate tools, and even then, the metrics told you that performance had degraded without telling you why.</p> <p>Enhanced observability closes three gaps. It isolates a single bad accelerator across hundreds or thousands of GPUs whether through XID errors, ECC memory faults, or thermal throttling, so you spend time on the fix instead of the search.</p> <p>Together, these capabilities reduce troubleshooting time and help you keep expensive accelerators busy.</p> <h2 id="how-it-works">How it works</h2> <p>Each capability targets a different layer of a distributed training job.</p> <h3 id="default-and-advanced-metrics">Default and advanced metrics</h3> <p>Enhanced observability collects telemetry at two levels:</p> <p><strong>Default metrics</strong> (enabled automatically) cover cluster health, CPU/disk/OS-level usage, GPU utilization and temperature, and per-job resource utilization. These metrics are collected every 30 seconds and sent to your chosen metric store with approximately 1–2% CPU overhead and 250–300 MB of memory.</p> <p><strong>Advanced metrics</strong> (opt-in) provide deeper visibility:</p> <ul> <li><strong>Full node exporter suite:</strong> comprehensive host-level metrics.</li> <li><strong>All NVIDIA GPU DCGM and AWS Neuron metrics:</strong> detailed accelerator telemetry including clock speeds, power, PCIe activity, and error indicators.</li> <li><strong>Elastic Fabric Adapter (EFA) metrics:</strong> network interface throughput, drops, retransmissions, and errors.</li> </ul> <p>Advanced metrics are optional because they increase the volume of telemetry sent to your metric store, which has a cost implication. Most customers running small jobs on small clusters may not need them, but they are essential for long-running training jobs on large clusters where patterns emerge over time.</p> <h3 id="metric-stores">Metric stores</h3> <p>You can choose between two metric stores:</p> <ul> <li><strong>Amazon CloudWatch</strong> (new default) provides native monitoring through CloudWatch SageMaker Insights, so you can visualize metrics without leaving the AWS Management Console.</li> <li><strong>Amazon Managed Service for Prometheus</strong> for teams that standardize on Prometheus and Grafana for observability.</li> </ul> <p>Previously, only Prometheus was supported. CloudWatch is now the default, offering lower pricing compared to Amazon Managed Service Prometheus and tighter integration with the SageMaker console.</p> <h3 id="diagnostic-summary">Diagnostic summary</h3> <p>The diagnostic summary uses two complementary approaches to surface issues across your HyperPod cluster.</p> <p><strong>GPU outlier detection</strong> uses CloudWatch’s <em>DetectGroupOutliers</em> API to compare GPUs within a training job across utilization, memory usage, temperature, power draw, and SM clock frequency. When a GPU deviates significantly from its peers, it appears as an outlier card in the diagnostic summary. This works for both EKS and Slurm jobs.</p> <p><strong>Static diagnostic rules</strong> evaluate known failure patterns and organize detected issues by type:</p> <ul> <li><strong>Hardware:</strong> GPU ECC memory errors, XID faults, accelerator failures.</li> <li><strong>Network:</strong> EFA throughput degradation, packet drops, retransmissions.</li> <li><strong>Throttling:</strong> GPU clock throttling because of heat, board, or reliability limits.</li> <li><strong>Overheating:</strong> GPU temperatures exceeding configured limits.</li> </ul> <p>Each detected issue identifies the affected job, cluster, node, accelerator, and rank.</p> <h2 id="how-to-get-started">How to get started</h2> <p>Enhanced observability is delivered as an add-on for EKS-orchestrated clusters and as an extension for Slurm-orchestrated clusters, which means you control who can install or change it through AWS Identity and Access Management (IAM).</p> <h3 id="step-1-create-a-sagemaker-hyperpod-cluster">Step 1: Create a SageMaker HyperPod cluster</h3> <p>If you already have a cluster, you can use it. To create a new SageMaker HyperPod cluster, you can follow the <a href="https://awslabs.github.io/ai-on-sagemaker-hyperpod/" target="_blank" rel="noopener">step-by-step outline</a>. In general, the steps are as follows:</p> <ol type="1"> <li>On the Amazon SageMaker AI console, choose <strong>HyperPod clusters</strong> in the navigation pane.</li> <li>Choose <strong>Create HyperPod</strong> cluster and <strong>Orchestrated on Amazon E