1 tiếng nói · chưa gắn nhãn trưởng lab · 1 nền tảng · chưa có số tương tác · Nguồn crawl
A
Aryaman Kothari 1 bài viết · AWS Machine Learning Blog Nguyên văn nguồn Chi tiết nguồn
<p>Once a model is in production, the questions change. Teams want to know which configuration the model should run on now that traffic is real, how to fit more models onto GPUs they already pay for, what to do when the next model is too large for one machine, why requests wait in one Region while GPUs sit idle in another, and why latency rose when nobody changed anything. This post takes those questions in the order teams usually meet them.</p> <p>For HyperPod Inference, it listed additional routing strategies, broader support for model serving frameworks, and expanded multi-Region networking. The Inference Gateway delivers the routing, and managed NVIDIA Dynamo is a new serving framework. For managed endpoints, the roadmap pointed to deeper integration with the rest of the AWS AI landscape. Inference recommendations now run from Amazon SageMaker AI Studio, the SageMaker Python SDK, and the coding agents engineers already use, and endpoints can now route requests across Regions and accounts.</p> <p>To keep each section concrete, we follow one hypothetical team through the blog post. In the spring, they launched a customer support assistant on a 70B-parameter model behind a single SageMaker AI endpoint, using the launches from part 1. By the fall, they need to support more product lines, adopt a larger reasoning model, serve customers in a second Region, and keep their GPU bill flat.</p> <h2 id="the-launches-at-a-glance">The launches at a glance</h2> <div style="width: 810px" class="wp-caption alignnone"> <a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/19/ML-22497-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/19/ML-22497-1.png" alt="Five inference stages in this post and the launches grouped under each" width="800"></a> <p class="wp-caption-text">Figure 1: The five stages in this post and the launches in each</p> </div> <h2 id="choose-the-right-configuration-from-wherever-you-work">Choose the right configuration from wherever you work</h2> <p>The team picked its first configuration from a benchmark run in April. Since then, they have added four LoRA (Low-Rank Adaptation) adapters, one for each product line, and the product team has set a target of 500 milliseconds for time to first token (TTFT) at peak load. The April configuration was never tested against either change. The engineers who will re-run the analysis work in notebooks and coding agents, and few of them use the console.</p> <p>Inference recommendations launched in April as an API. Four updates since then make it available in more of the places teams work, and let it answer two harder questions: how many adapters an endpoint can carry, and how much concurrency it can take before it misses a latency target. A new agent skill brings the same expertise into coding agents.</p> <h3 id="inference-recommendations-in-sagemaker-ai-studio-july-2026">Inference recommendations in SageMaker AI Studio (July 2026)</h3> <p><em>Blog:</em> <a href="https://aws.amazon.com/blogs/machine-learning/launching-ui-for-generative-ai-inference-recommendations-in-amazon-sagemaker-ai/" target="_blank" rel="noopener">Launching UI for generative AI inference recommendations in Amazon SageMaker AI</a></p> <p>The April launch automated instance, container, and optimization selection through APIs. Inference recommendations are now also available in SageMaker AI Studio as a guided, low-code workflow. In Studio, choose <strong>Jobs</strong>, then <strong>Inference optimization</strong>. Choose a use-case profile (Interact, Generate, Summarize, or Custom), an optimization goal (minimize latency, maximize throughput, or minimize cost), and a model from SageMaker JumpStart, Amazon Simple Storage Service (Amazon S3), the Model Registry, or an existing SageMaker model.</p> <p>SageMaker AI benchmarks candidate configurations on real GPU infrastructure using NVIDIA AIPerf and applies techniques that match the goal, such as speculative decoding for throughput or kernel tuning for latency. You compare the ranked results by TTFT, inter-token latency, throughput, and cost, and then deploy the chosen configuration to a real-time endpoint from Studio.</p> <h3 id="inference-recommendations-in-the-sagemaker-python-sdk-v3-august-2026">Inference recommendations in the SageMaker Python SDK v3 (August 2026)</h3> <p><em>Blog: <a href="https://aws.amazon.com/blogs/machine-learning/llm-optimization-integration-for-amazon-sagemaker-python-sdk/" target="_blank" rel="noopener">LLM optimization integration for Amazon SageMaker Python SDK</a></em></p> <p>Benchmarking and recommendations previously required SageMaker AI Studio or direct Boto3 calls. They are now part of the SageMaker Python SDK v3 (version 3.17.0 and later), the same interface teams use for training and deployment, so the workflow fits into notebooks, CI pipelines, and MLOps workflows. You can benchmark a live endpoint against your workload traffic, generate ranked recommendations for a latency, throughput, or cost goal, and deploy the top-ranked configuration to a real-time endpoint in a single call.</p> <p>The SDK also supports longer workflows. You can run recommendation jobs in parallel to compare serving frameworks such as LMI and vLLM, and you can resume from a completed job so that deployment happens in a different notebook or pipeline stage without re-running the benchmarks.</p> <h3 id="lora-adapter-support-in-inference-recommendations-september-2026">LoRA adapter support in inference recommendations (September 2026)</h3> <p><em>Developer guide</em>: <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/generative-ai-inference-recommendations-adapters.html" target="_blank" rel="noopener">Get recommendations for models with LoRA adapters</a></p> <p>Many teams use LoRA adapters to specialize one foundation model for several use cases. Each adapter kept in GPU memory reduces the nu