1 tiếng nói · chưa gắn nhãn trưởng lab · 1 nền tảng · chưa có số tương tác · Nguồn crawl
H
Hazim Qudah 1 bài viết · AWS Machine Learning Blog Nguyên văn nguồn Chi tiết nguồn
<p>Every inference endpoint is well tuned the day it goes into production. Teams choose an instance type, serving configuration, and scaling settings that suit the model and its traffic at launch. With the endpoint serving smoothly, they understandably move on to the next priority. The workload, however, rarely stays still. Traffic patterns evolve, prompts grow longer, serving engines such as vLLM ship new releases, SageMaker inference develops new optimization techniques, and new fine-tuned checkpoints change the resources your model needs. Over time, a well-tuned endpoint quietly drifts. Latency and time to first token edge upward. A configuration that was never quite ideal keeps missing available performance gains. A GPU provisioned for demand that never materialized sits idle week after week. Because nothing has failed, no alarm sounds, and the drift goes unnoticed until users feel it or the bill reflects it.</p> <p><a href="https://aws.amazon.com/sagemaker/ai/" target="_blank" rel="noopener">Amazon SageMaker AI</a> now monitors your generative AI inference endpoints and tells you when a change could improve performance or lower cost by using <a href="https://aws.amazon.com/blogs/machine-learning/amazon-sagemaker-ai-now-supports-optimized-generative-ai-inference-recommendations/" target="_blank" rel="noopener">Generative AI optimized inference recommendations</a> feature with Agentic experience in <a href="https://aws.amazon.com/sagemaker/ai/studio/" target="_blank" rel="noopener">Amazon SageMaker Studio</a>. Monitoring is on by default for SageMaker AI inference endpoints using vLLM serving containers and GPU instances, with no setup required, and you can opt out of endpoints. SageMaker AI looks only at operational signals: latency, GPU utilization, and the endpoint’s serving configuration. It does not read your requests or responses. From those signals, it surfaces three kinds of findings:</p> <ul> <li><strong>Latency drift:</strong> A week-over-week rise in median (p50) end-to-end latency or time to first token.</li> <li><strong>Unused capacity:</strong> GPUs you pay for that sit mostly idle.</li> <li><strong>Configuration gaps:</strong> A serving container that’s missing newer performance improvements or security patches, or an optimization such as speculative decoding or GPU kernel tuning that your model supports but the endpoint doesn’t use.</li> </ul> <p>SageMaker AI sends a finding through AWS Health, with one notification per endpoint that covers that endpoint’s findings. When a notification arrives, you can act on it in one place. It opens the SageMaker AI agentic experience in Amazon SageMaker Studio with the endpoint, and its recent traffic already filled in. From there, you run inference recommendations to benchmark your current configuration against alternatives and compare them side by side on response time, throughput, and cost per million tokens. That turns a judgment call into a decision backed by data. Benchmarks run on separate compute, and nothing changes on your endpoint until you choose to apply a result.</p> <p>In this post, we show how SageMaker AI monitors your endpoints and how to read the notifications it sends through AWS Health. We then show how to go from a notification to a benchmarked decision in SageMaker Studio. We walk through three example scenarios, show how to apply or dismiss a recommendation safely, and explain how to opt an endpoint out of monitoring.</p> <h2 id="solution-overview">Solution overview</h2> <p>SageMaker AI inference recommendations follow a four-step loop: observe, notify, recommend, and decide. Each step is designed so that you stay in control, and your production endpoint isn’t changed without your explicit action.</p> <ol type="1"> <li><strong>Observe –</strong> Monitoring of latency and capacity usage is on by default for eligible endpoints. It doesn’t read your requests or responses, and it carries no additional GPU cost.</li> <li><strong>Notify –</strong> When there is something worth acting on, you receive a plain language notification in AWS Health, plus email, explaining what was observed, why it matters, and a recommended next step.</li> <li><strong>Recommend (opt in) –</strong> One selection opens the SageMaker AI agentic experience in SageMaker Studio, prefilled with the endpoint and its observed traffic, so you can benchmark alternatives on separate compute.</li> <li><strong>Decide –</strong> Compare response time, throughput, and cost side by side. Production is not changed until you apply a result.</li> </ol> <p>The feature produces a few types of findings. The following table summarizes what triggers each one and what gets tested when you choose to benchmark.</p> <table border="1px" width="100%" cellpadding="10px"> <tbody> <tr> <td><strong>Finding</strong></td> <td><strong>What triggers it</strong></td> <td><strong>What a benchmark tests</strong></td> </tr> <tr> <td>Latency drift</td> <td>Response time moves relative to the endpoint’s week-over-week trends</td> <td>Tensor parallelism settings and alternate instance types or sizes</td> </tr> <tr> <td>Underutilized GPUs</td> <td>GPU utilization stays well below capacity over the window</td> <td>Cheaper instance types and smaller sizes at observed peak traffic</td> </tr> <tr> <td>Speculative decoding not enabled</td> <td>Model and container support it, but it is off (configuration based)</td> <td>Current configuration compared to speculative decoding enabled</td> </tr> <tr> <td>Serving container</td> <td>The endpoint’s serving container has a newer version with a performance improvement and security patches for critical and high finding Common Vulnerabilities and Exposures (CVEs).</td> <td>Current compared to updated container</td> </tr> <tr> <td>Untuned GPU kernels</td> <td>Kernel configuration is not tuned for the model and hardware</td> <td>Default compared to tuned kernels</td> </tr> </tbody> </table> <h2 id="prerequisites">Prerequisite