AllyshipContent Radar

Hands-on 278

Công cụ, thử nghiệm và hướng dẫn thực hành

Crawl live-1795090500000 · 19:15 19 thg 11, 2026 · Xuất bản 19:35 19 thg 11, 2026

13

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

58.9

Hướng dẫn Roy Allela

<p>Large training jobs can fail even when every node looks healthy. A GPU with a subtle fault can pass its stress tests and still return wrong values. When a job stalls on an NCCL timeout, it keeps its GPUs, and Slurm still shows it as running. If a device plugin crashes on one node, the job fails with an error that never mentions the plugin. Teams usually find these problems only after the loss diverges or the job fails, and by then the compute is spent. Each of these failures lowers goodput, the share of your cluster’s time that goes to useful training. Today, we’re introducing workload recovery and infrastructure checks on Amazon SageMaker HyperPod that catch these failures sooner and raise the goodput of your training workloads.</p> <p>SageMaker HyperPod already handles nodes that report hardware faults. The health monitoring agent (HMA) watches each node for GPU errors, deep health checks test nodes during provisioning, and automatic node recovery reboots or replaces faulty hardware. These features look at one node’s hardware at a time. They can’t tell whether the GPUs train correctly, whether a job is still making progress, or whether the Kubernetes components a job depends on are working.</p> <p>In this post, we walk through the three new capabilities that catch these failures. For each one, we show how to set it up and what it reports when it finds a problem:</p> <ol type="1"> <li><code>MultiNodeTrainingValidation</code> is a new deep health check for HyperPod clusters orchestrated by Amazon Elastic Kubernetes Service (Amazon EKS). When HyperPod provisions an instance, or when you run the check on demand, it trains a small model on the instance and then across instances of the same type. The instance fails if the loss becomes NaN or infinite, or training doesn’t finish in time.</li> <li>The HyperPod Kubernetes Resource Monitoring Agent watches the Kubernetes layer of an EKS cluster for problems such as crash-looping device plugins, missing DaemonSets, and tainted nodes. When a job fails, it reports the pods and nodes involved as Kubernetes events and Amazon CloudWatch logs.</li> <li>Hung job detection finds Slurm jobs that have stopped making progress. An agent on each node watches the job’s output and GPU activity, or log patterns that you define. It writes an event to CloudWatch and can cancel the job so that Slurm requeues it.</li> </ol> <h2 id="why-node-health-alone-isnt-enough">Why node health alone isn’t enough</h2> <p>The existing HyperPod resiliency features check a single node at a time. The health monitoring agent (HMA) watches each node for GPU errors such as Xid and NVLink faults, Elastic Fabric Adapter (EFA) errors, and kernel errors. The InstanceStress deep health check runs DCGM level 4 diagnostics and tests each instance’s GPUs, GPU memory, NVLink, and EFA devices. The InstanceConnectivity check runs NCCL tests across GPU instances, or NCCOM tests on AWS Trainium. Together, these confirm that each node’s hardware works. They don’t show whether the GPUs compute correctly, whether a job is still making progress, or whether the cluster’s Kubernetes components are healthy. Three kinds of failure get past them:</p> <ul> <li><strong>Numerical faults –</strong>&nbsp;A GPU can pass every hardware test and still compute wrong results. The first sign is often NaN or infinite loss in a production run. Meta attributed 6 of the 419 unexpected interruptions in its 54-day Llama 3 405B pre-training run to silent data corruption (<a href="https://arxiv.org/abs/2407.21783" target="_blank" rel="noopener">The Llama 3 Herd of Models</a>).</li> <li><strong>Hung jobs –</strong>&nbsp;An NCCL timeout, a stalled data pipeline, or a deadlock can stop a job without ending it. The job keeps its GPUs and their memory, and Slurm reports it as running until someone checks the loss curve or the log.</li> <li><strong>Cluster-level failures –</strong>&nbsp;Some failures happen in Kubernetes, not in a node’s hardware. A device plugin crash-loops, a required DaemonSet never starts on a new node, or ranks on different nodes fall out of step in an NCCL collective. HMA checks each node on its own, so it doesn’t see these problems. When the job fails, the error often doesn’t say which pod or node caused it.</li> </ul> <h2 id="solution-overview">Solution overview</h2> <p>Each capability covers a different point in a training run. <code>MultiNodeTrainingValidation</code> runs when instances are provisioned, or on demand, and confirms that their GPUs train correctly. Hung job detection runs alongside each Slurm job and catches a job that has stopped making progress. The HyperPod Kubernetes Resource Monitoring Agent watches the cluster continuously and, when a job fails, reports the pods and nodes that were involved. The following table summarizes the three.</p> <table border="1px" width="100%" cellpadding="10px"> <tbody> <tr> <td><strong>Capability</strong></td> <td><strong>What it catches</strong></td> <td><strong>When it runs</strong></td> <td><strong>Available on</strong></td> </tr> <tr> <td><code>MultiNodeTrainingValidation</code> deep health check</td> <td>NaN or infinite loss, or training that doesn’t finish in time, on GPUs that pass hardware checks</td> <td>When instances are created or updated, or on demand</td> <td>EKS clusters, on supported EFA-enabled NVIDIA GPU instances</td> </tr> <tr> <td>HyperPod Kubernetes Resource Monitoring Agent</td> <td>Node condition changes, pod and DaemonSet failures, and failed training jobs, with the pods and nodes involved. NCCL communication state for jobs that opt in</td> <td>Continuously</td> <td>EKS clusters</td> </tr> <tr> <td>Hung job detection</td> <td>Jobs that stop making progress but don’t exit</td> <td>While each job runs</td> <td>Slurm clusters</td> </tr> </tbody> </table> <p>Hung job detection is new for Slurm clusters. On EKS clusters, HyperPod already detects hung jobs that run through the <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/sag

AWS Machine Learning Blog · 19 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
14

Introducing the Amazon SageMaker JumpStart Model Leaderboard

58.9

Hướng dẫn Dan Ferguson

<p>Amazon SageMaker JumpStart gives you hundreds of open-weight foundation models you can deploy quickly. That choice is powerful, but it also creates a real question: Which model should you start with? Comparing models usually means reading model cards one at a time and hunting for benchmark scores that different sources published under different conditions. Then you still must guess how each model will perform, and what it will cost, for your use case.</p> <p>Today we are launching the <strong>Amazon SageMaker JumpStart Model Leaderboard</strong>, a new way to compare <a href="https://console.aws.amazon.com/sagemaker/home#/launch?source=jumpstart&amp;page=jumpstart" target="_blank" rel="noopener">JumpStart models</a> side by side on quality, speed, and cost, directly in Amazon SageMaker Studio. In this post, we explain what the leaderboard shows, define the key terms in plain language, and walk through how to go from a shortlist to a deployed model.</p> <h2 id="what-is-the-jumpstart-model-leaderboard">What is the JumpStart Model Leaderboard?</h2> <p>The leaderboard is a new view of the JumpStart base models page in SageMaker Studio, right next to the Categories and All models views. It uses the same search bar and filters you already know, and it opens on a leaderboard of large language models (LLMs). You see rankings immediately without answering setup questions first. For each model, it shows two kinds of information:</p> <ol type="1"> <li><strong>Quality scores</strong> on three public benchmarks (BIG-Bench Hard, GPQA Diamond, and MMLU-Pro) that AWS runs itself, using the same evaluation setup for every model so the numbers are comparable.</li> <li><strong>Performance and cost metrics</strong> that AWS measures by sending a standardized test workload to an endpoint using each JumpStart deployment configuration, so you see how fast a model responds and what it costs to run.</li> </ol> <p>Because both kinds of numbers come from the same place and the same method, you can make a like-for-like comparison instead of stitching together data from different websites. At launch, the leaderboard covers <strong>70+ models</strong> from providers including Meta, Google, Qwen, IBM, and Arcee, with more added over time.</p> <p>The following screenshot shows the leaderboard in SageMaker Studio, with benchmark scores, performance metrics, and estimated cost for each model.</p> <div style="width: 810px" class="wp-caption alignnone"> <a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/18/ML-22496-1-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/18/ML-22496-1-1.png" alt="SageMaker JumpStart Model Leaderboard in SageMaker Studio showing benchmark scores, performance metrics, and estimated cost per model" width="800"></a> <p class="wp-caption-text">Figure 1: Leaderboard overview in SageMaker Studio</p> </div> <h2 id="key-concepts">Key concepts</h2> <p>This section defines the terms the leaderboard uses: the quality benchmarks, the use case profiles and optimization goals that shape the performance numbers, and the performance metrics themselves.</p> <h3 id="quality-benchmarks">Quality benchmarks</h3> <p>A benchmark is a standardized test: a fixed set of questions with known answers. Running every model through the same test gives you a score you can compare. At launch, the leaderboard reports three reasoning benchmarks for LLMs:</p> <table border="1px" width="100%" cellpadding="10px"> <tbody> <tr> <td><strong>Benchmark</strong></td> <td><strong>What it measures, in plain terms</strong></td> </tr> <tr> <td><strong>BIG-Bench Hard (BBH)</strong></td> <td>A set of challenging tasks that require multi-step reasoning, such as logical deduction, tracking objects, and multi-step arithmetic.</td> </tr> <tr> <td><strong>GPQA Diamond</strong></td> <td>Graduate-level science questions that are difficult to answer by searching the web. A good signal of deep reasoning.</td> </tr> <tr> <td><strong>MMLU-Pro</strong></td> <td>Broad knowledge and reasoning across many subjects, such as science, law, and engineering, using harder multiple-choice questions.</td> </tr> </tbody> </table> <p>Higher is better for all three. The leaderboard shows each benchmark separately rather than blending them into one number. That keeps the results transparent: You can weigh the benchmarks that matter for your application and check each score against the public definition of the benchmark.</p> <h3 id="use-case-profiles-and-optimization-goals">Use case profiles and optimization goals</h3> <p>Speed and cost depend on your workload and on how the model is deployed. Two menus above the leaderboard control which performance and cost numbers you see:</p> <ol type="1"> <li><strong>Use case profile.</strong> Select the profile closest to your workload from the View benchmarks for your use case menu. The default is Max Context. The profile changes only the performance and cost metrics. Quality benchmark scores are the same for every profile.</li> <li><strong>Optimization goal.</strong> For each profile, a model can have several deployment configurations, and the leaderboard shows metrics for one at a time. Choose which one with the Optimized for menu. The default is Balanced.</li> </ol> <p>Each optimization goal picks a different configuration:</p> <table border="1px" width="100%" cellpadding="10px"> <tbody> <tr> <td><strong>Optimization goal</strong></td> <td><strong>Configuration shown</strong></td> </tr> <tr> <td><strong>Balanced</strong></td> <td>The configuration with the lowest estimated cost per token. Because cost per token depends on both instance price and throughput, this balances what you pay per hour against how fast the instance generates tokens.</td> </tr> <tr> <td><strong>Cost optimized</strong></td> <td>The configuration with the lowest hourly instance price. Latency can be higher and throughput lower

AWS Machine Learning Blog · 19 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
15

How to turn your Grok Bot into a top 1% AI agent content team (research, scripts, editing etc)

58.8

Hướng dẫn gregisenberg

How to turn your Grok Bot into a top 1% AI agent content team (research, scripts, editing etc) You give your Grok Bot the creators you love, hooks you’ve saved, fav newsletters, and it’ll find what’s working, write script, cut your edit and publish the video: https://youtu.be/vKobP6DkC1E?si=0_Xo9L_qjNXKjfrT Media

X · 19 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
16

I think a lot of people aren't using Grok Bot to its full potential, because knowing how to automate things is actually quite a difficult skill to learn. I wrote this post to break down how I approach

58.7

Hướng dẫn poteto

I think a lot of people aren't using Grok Bot to its full potential, because knowing how to automate things is actually quite a difficult skill to learn. I wrote this post to break down how I approach automating anything with Grok Bot! https://x.com/i/article/2122921713864171520

X · 19 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
17

Private certificate authority support in Amazon Bedrock AgentCore Identity

58.7

Hướng dẫn Aastha Paul

<p>Many teams that build applications self-host their identity provider (IdP), such as Keycloak or PingFederate, inside a private network. These IdPs usually present a TLS server certificate from a private certificate authority (CA). With private CA support in <a href="https://aws.amazon.com/bedrock/agentcore/" target="_blank" rel="noopener">AgentCore Identity</a>, a capability of Amazon Bedrock AgentCore, your agents can now connect to these IdPs while trusting the private CA that your organization already uses.</p> <p>AgentCore Identity now supports private CA certificates for private identity providers. You store the PEM-encoded CA certificate in AWS Secrets Manager or Amazon Simple Storage Service (Amazon S3) and <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/identity-outbound-credential-provider.html" target="_blank" rel="noopener">reference it from your credential provider</a>. The CA certificate contains only the public certificate, not its private key. AgentCore Identity uses it to validate the server certificate presented by your IdP’s discovery, JSON Web Key Set (JWKS), and token endpoints. The CA certificate must be a root or intermediate CA certificate.</p> <p>In this post, we show you how to issue an IdP certificate with AWS Private Certificate Authority. We then configure an <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/identity-outbound-credential-provider.html" target="_blank" rel="noopener">AgentCore Identity credential provider</a> to trust it and rotate the CA while minimizing downtime. We also show how to protect the CA certificate so that only your IdP team can change it. Although AgentCore Identity supports any private CA, including enterprise, third-party, and self-signed root CAs, the walkthrough in this post uses AWS Private CA and Keycloak as the IdP.</p> <h2 id="solution-overview">Solution overview</h2> <p>Consider an agent that needs to call an internal wiki API protected by your Keycloak IdP. The agent runs on <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/agents-tools-runtime.html" target="_blank" rel="noopener">AgentCore Runtime</a>, a capability of Amazon Bedrock AgentCore, which hosts your agent code. To get an OAuth 2.0 access token from Keycloak, the agent uses an AgentCore Identity credential provider, which holds the IdP connection details and client credentials.</p> <p>The following diagram shows this setup. Keycloak runs in your virtual private cloud (VPC) behind an internal Application Load Balancer. The load balancer presents an <a href="https://aws.amazon.com/certificate-manager/" target="_blank" rel="noopener">AWS Certificate Manager (ACM)</a> private certificate signed by AWS Private CA. The CA certificate that AgentCore Identity trusts is stored in Secrets Manager or Amazon S3.</p> <p>AgentCore Identity reaches the IdP privately through <a href="https://aws.amazon.com/vpc/lattice/" target="_blank" rel="noopener">Amazon VPC Lattice</a>, which connects to your VPC through a <a href="https://docs.aws.amazon.com/vpc-lattice/latest/ug/resource-gateway.html" target="_blank" rel="noopener">resource gateway</a>, an entry point for VPC Lattice traffic into your VPC. VPC Lattice is required for private CA support. If you choose the managed option, AgentCore Identity creates and manages the VPC Lattice resources for you.</p> <div style="width: 810px" class="wp-caption alignnone"> <a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/18/ML-22407-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/18/ML-22407-1.png" alt="AgentCore Identity reaches a Keycloak IdP through VPC Lattice and validates its server certificate against a private CA." width="800"></a> <p class="wp-caption-text">Figure 1: Connecting AgentCore Identity to a self-hosted IdP that uses a private CA</p> </div> <p>The flow works as follows:</p> <ol type="1"> <li>You store the CA certificate in Secrets Manager or Amazon S3.</li> <li>You create a credential provider that points to your IdP and to the CA certificate.</li> <li>AgentCore Identity reads the CA certificate and checks that it’s a valid CA certificate.</li> <li>Your agent requests an access token from AgentCore Identity.</li> <li>AgentCore Identity connects to the IdP through VPC Lattice and checks the load balancer certificate against your CA.</li> <li>The IdP issues the token and AgentCore Identity returns it to your agent.</li> </ol> <p>Agents that use machine-to-machine, on-behalf-of, and user-delegated flows can also use the private CA without changing their code.</p> <h2 id="prerequisites">Prerequisites</h2> <p>To follow this walkthrough, you need:</p> <ul> <li>An <a href="https://aws.amazon.com/resources/create-account/" target="_blank" rel="noopener">AWS account</a> with access to the <a href="https://console.aws.amazon.com/bedrock-agentcore" target="_blank" rel="noopener">AgentCore console</a>, in a <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/agentcore-regions.html" target="_blank" rel="noopener">supported AWS Region</a> where AgentCore Identity supports private IdPs.</li> <li>A self-hosted, OpenID Connect (OIDC) compliant IdP that runs in your VPC, with an OAuth client registered for your agent. The networking requirements, including subnets, security groups, and a private hosted zone for the IdP hostname, are described in <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/identity-private-idp.html#private-idp-prerequisites" target="_blank" rel="noopener">Connect to private identity providers</a>.</li> <li>The latest version of the <a href="https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html" target="_blank" rel="noopener">AWS Command Line Interface (AWS CLI)</a> and <a href="https://openssl-library.org/" target="_blank" rel="noopener">OpenSSL</a>.</li> <li

AWS Machine Learning Blog · 19 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
13–18 / 179
Trang 3 / 30