AllyshipContent Radar

Deep dives 738

Paper và bài viết kỹ thuật

Crawl live-1795068900000 · 13:22 19 thg 11, 2026 · Xuất bản 13:30 19 thg 11, 2026

Đánh giá

How Airbnb uses cost per task to operate coding agents on Amazon Bedrock

<p>Amazon Bedrock provides access to foundation models from multiple providers through a common API, with AWS managing the model infrastructure. It integrates with AWS access controls and monitoring, and processes prompts, code, and model responses within AWS.</p> <p>As coding agents move into production, changes in models and usage can affect latency, capacity, and cost. Amazon Bedrock runtime metrics in Amazon CloudWatch help teams investigate latency increases, throttling, and changes in token consumption. Evaluations, or evals, complement these operational metrics by assessing the quality of an agent’s work on a fixed set of representative tasks.</p> <p>In this post, we show how Airbnb evaluates models for coding agents and monitors their use on Amazon Bedrock. We explain how cost per task guides model selection and how production data helps us revisit those choices as usage changes. We also describe the runtime signals used to investigate latency, throttling, and spend.</p> <p>At Airbnb, thousands of engineers and engineering-adjacent people use AirChat, our internal tool for running coding agents, every week. The agents send their model requests through our large language model (LLM) gateway, which runs on Amazon Bedrock. Bedrock processes Airbnb’s code, prompts, and model responses within AWS. Eight new models came out in about five weeks. By mid-October, about two thirds of the requests our engineers sent through coding agents went to those new models.</p> <p>Each release can change output tokens, cost, latency and capacity at once, and the benchmark charts published with a launch don’t show how a model handles our engineers’ work. So we run two systems side by side. Before a rollout, we run evals on our own tasks to measure each model’s score and cost per task. Cost per task is the list-price cost of all the tokens an agent uses to finish one task.</p> <p>After a rollout, observability shows us what changed in production. When production usage doesn’t match the eval results, we find out why.</p> <h2 id="solution-overview">Solution overview</h2> <p>You can diagnose most production problems with a few signals in the <code>AWS/Bedrock</code> namespace in CloudWatch:</p> <ul> <li>Latency: <code>InvocationLatency</code>, <code>TimeToFirstToken</code>, <code>OutputTokenCount</code>, and output tokens per second (OTPS), which you derive.</li> <li>Throttling: <code>InvocationThrottles</code>, plus the tokens each request reserves against your quota, which you publish yourself. Bedrock also publishes <code>EstimatedTPMQuotaUsage</code>, which measures your estimated tokens per minute (TPM) quota consumption, including cache write tokens and output burndown multipliers, across all inference APIs (<code>Converse</code>, <code>InvokeModel</code>, <code>ConverseStream</code>, and <code>InvokeModelWithResponseStream</code>). You can use this metric to set proactive alarms before reaching your quota limit, track your quota consumption across your models, and request further quota increases before usage is rate limited.</li> <li>Cost: <code>InputTokenCount</code> (uncached input), <code>CacheReadInputTokenCount</code>, <code>CacheWriteInputTokenCount</code> and <code>OutputTokenCount</code>, each priced at its own rate. Cache reads cost less than uncached input, and cache writes can cost more depending on the model. A lower cache hit rate can increase cost per request.</li> </ul> <p>On top of these metrics, we add cost per task. Our evals measure a score and a cost per task for every model and every reasoning effort we allow. Reasoning effort is a setting that controls how much a model reasons before it responds. We use the results to choose each default, which is the model and effort that our coding tools use unless an engineer picks another, and each effort ceiling, which is the highest effort we allow for a model.</p> <h2 id="architecture">Architecture</h2> <p>Engineers run AirChat on their AirDev workspaces, which are personal development machines in the cloud. They can also hand a task to AirChat Remote, which runs cloud agents in separate cloud workspaces while the engineer does something else. Both paths go through our LLM gateway to Amazon Bedrock.</p> <p>Figure 1 shows how AirChat and AirChat Remote reach Amazon Bedrock through our gateway. It also shows how evals and production monitoring guide model defaults and effort ceilings.</p> <div style="width: 810px" class="wp-caption alignnone"> <a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/18/ML-22452-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/18/ML-22452-1.png" alt="AirChat and AirChat Remote reach Amazon Bedrock through Airbnb’s gateway, with CloudWatch metrics and eval results setting model defaults and effort ceilings" width="800"></a> <p class="wp-caption-text">Figure 1: Coding agents and cloud agents call Amazon Bedrock through our LLM gateway. Our usage records and CloudWatch metrics show real use, and we use eval results to set defaults and effort ceilings</p> </div> <p>The gateway records the model, client, team, and user for each request. We use these records only in aggregate, for cost and capacity planning. CloudWatch supplies token counts, latency, errors, and throttles. We compare those production signals with our eval scores and cost per task to decide when defaults or effort ceilings need to change.</p> <h2 id="prerequisites">Prerequisites</h2> <p>On the AWS side, you need:</p> <ul> <li>Access to the models in Amazon Bedrock you plan to run and evaluate. For model availability by AWS Region, see <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/models-region-compatibility.html" target="_blank" rel="noopener">Regional availability by models</a> in the Amazon Bedrock User Guide.</li> <li>Streaming calls through <code>ConverseStream</code> or <code>InvokeModelWithResponseStrea

  • </p> <p>In this post, we show how Airbnb evaluates models for coding agents and monitors their use on Amazon Bedrock.
  • png" alt="AirChat and AirChat Remote reach Amazon Bedrock through Airbnb’s gateway, with CloudWatch metrics and eval results setting model defaults and effort ceilings" width="800"></a> <p class="wp-caption-text">Figure 1: Coding agents and cloud agents call Amazon Bedrock through our LLM gateway.
  • Our usage records and CloudWatch metrics show real use, and we use eval results to set defaults and effort ceilings</p> </div> <p>The gateway records the model, client, team, and user for each request.
Nguyên văn nguồn
Chi tiết nguồn
Kỹ thuật

Show HN: Re-SiON – 40kb software synthesizer library that plays plaintext files

Hi HN! Re-SiON is a re-implementation of an old software synthesizer created for Adobe Flash. It's incredibly tiny (at ~40KB) and runs on the web and desktop platforms. It plays easily editable plaintext MML (Music Macro Language) files which are insanely tiny, many example songs only taking up a few kilobytes. The project also comes with a playground to play and edit MML in real time, and a converter that can convert from tracker formats and NES/SNES/GameBoy/Genesis music straight to MML. It's all open source under MIT, and the link has a bunch of example songs to listen to. Happy to answer any questions.

Nguyên văn nguồn
Chi tiết nguồn
Kiến trúc

Show HN: Reprise, a read-later app with narration on the neural engine for iOS

I've been working on this app for a little under a year now. I've gotten PocketTTS working with voice cloning running on an iPhone's neural engine, at speeds where you can narrate an article that has 20 minutes of audio in under 20 seconds.

Nguyên văn nguồn
Chi tiết nguồn
Tác tử

Show HN: Dez – plaintext open source AI multimedia studio

Hi HN! Dez is a free open source way to interact with AIs across providers in a full-featured text editor. I really loved Zed's text threads feature, and after they removed it got inspired to revive it in this app. In this latest release, Dez acts as a sandboxed agent with multimedia focused tools. It can invoke other models to generate assets and call its bundled ffmpeg to edit them. Dez won't harm your computer, it has no way of doing so, and the ffmpeg command is sandboxed with Seatbelt on macOS and Landlock on linux. It's also built with privacy in mind, so it's designed not to send absolute file paths to providers that might reveal things like usernames, folder names, etc. Give it a try and let me know what you think!

Nguyên văn nguồn
Chi tiết nguồn
Tác tử

Private certificate authority support in Amazon Bedrock AgentCore Identity

<p>Many teams that build applications self-host their identity provider (IdP), such as Keycloak or PingFederate, inside a private network. These IdPs usually present a TLS server certificate from a private certificate authority (CA). With private CA support in <a href="https://aws.amazon.com/bedrock/agentcore/" target="_blank" rel="noopener">AgentCore Identity</a>, a capability of Amazon Bedrock AgentCore, your agents can now connect to these IdPs while trusting the private CA that your organization already uses.</p> <p>AgentCore Identity now supports private CA certificates for private identity providers. You store the PEM-encoded CA certificate in AWS Secrets Manager or Amazon Simple Storage Service (Amazon S3) and <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/identity-outbound-credential-provider.html" target="_blank" rel="noopener">reference it from your credential provider</a>. The CA certificate contains only the public certificate, not its private key. AgentCore Identity uses it to validate the server certificate presented by your IdP’s discovery, JSON Web Key Set (JWKS), and token endpoints. The CA certificate must be a root or intermediate CA certificate.</p> <p>In this post, we show you how to issue an IdP certificate with AWS Private Certificate Authority. We then configure an <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/identity-outbound-credential-provider.html" target="_blank" rel="noopener">AgentCore Identity credential provider</a> to trust it and rotate the CA while minimizing downtime. We also show how to protect the CA certificate so that only your IdP team can change it. Although AgentCore Identity supports any private CA, including enterprise, third-party, and self-signed root CAs, the walkthrough in this post uses AWS Private CA and Keycloak as the IdP.</p> <h2 id="solution-overview">Solution overview</h2> <p>Consider an agent that needs to call an internal wiki API protected by your Keycloak IdP. The agent runs on <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/agents-tools-runtime.html" target="_blank" rel="noopener">AgentCore Runtime</a>, a capability of Amazon Bedrock AgentCore, which hosts your agent code. To get an OAuth 2.0 access token from Keycloak, the agent uses an AgentCore Identity credential provider, which holds the IdP connection details and client credentials.</p> <p>The following diagram shows this setup. Keycloak runs in your virtual private cloud (VPC) behind an internal Application Load Balancer. The load balancer presents an <a href="https://aws.amazon.com/certificate-manager/" target="_blank" rel="noopener">AWS Certificate Manager (ACM)</a> private certificate signed by AWS Private CA. The CA certificate that AgentCore Identity trusts is stored in Secrets Manager or Amazon S3.</p> <p>AgentCore Identity reaches the IdP privately through <a href="https://aws.amazon.com/vpc/lattice/" target="_blank" rel="noopener">Amazon VPC Lattice</a>, which connects to your VPC through a <a href="https://docs.aws.amazon.com/vpc-lattice/latest/ug/resource-gateway.html" target="_blank" rel="noopener">resource gateway</a>, an entry point for VPC Lattice traffic into your VPC. VPC Lattice is required for private CA support. If you choose the managed option, AgentCore Identity creates and manages the VPC Lattice resources for you.</p> <div style="width: 810px" class="wp-caption alignnone"> <a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/18/ML-22407-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/18/ML-22407-1.png" alt="AgentCore Identity reaches a Keycloak IdP through VPC Lattice and validates its server certificate against a private CA." width="800"></a> <p class="wp-caption-text">Figure 1: Connecting AgentCore Identity to a self-hosted IdP that uses a private CA</p> </div> <p>The flow works as follows:</p> <ol type="1"> <li>You store the CA certificate in Secrets Manager or Amazon S3.</li> <li>You create a credential provider that points to your IdP and to the CA certificate.</li> <li>AgentCore Identity reads the CA certificate and checks that it’s a valid CA certificate.</li> <li>Your agent requests an access token from AgentCore Identity.</li> <li>AgentCore Identity connects to the IdP through VPC Lattice and checks the load balancer certificate against your CA.</li> <li>The IdP issues the token and AgentCore Identity returns it to your agent.</li> </ol> <p>Agents that use machine-to-machine, on-behalf-of, and user-delegated flows can also use the private CA without changing their code.</p> <h2 id="prerequisites">Prerequisites</h2> <p>To follow this walkthrough, you need:</p> <ul> <li>An <a href="https://aws.amazon.com/resources/create-account/" target="_blank" rel="noopener">AWS account</a> with access to the <a href="https://console.aws.amazon.com/bedrock-agentcore" target="_blank" rel="noopener">AgentCore console</a>, in a <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/agentcore-regions.html" target="_blank" rel="noopener">supported AWS Region</a> where AgentCore Identity supports private IdPs.</li> <li>A self-hosted, OpenID Connect (OIDC) compliant IdP that runs in your VPC, with an OAuth client registered for your agent. The networking requirements, including subnets, security groups, and a private hosted zone for the IdP hostname, are described in <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/identity-private-idp.html#private-idp-prerequisites" target="_blank" rel="noopener">Connect to private identity providers</a>.</li> <li>The latest version of the <a href="https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html" target="_blank" rel="noopener">AWS Command Line Interface (AWS CLI)</a> and <a href="https://openssl-library.org/" target="_blank" rel="noopener">OpenSSL</a>.</li> <li

  • </p> <p>In this post, we show you how to issue an IdP certificate with AWS Private Certificate Authority.
Nguyên văn nguồn
Chi tiết nguồn
Tác tử

v0.24.0

<h2><a href="https://github.com/openai/openai-agents-python/compare/v0.23.1...v0.24.0">0.24.0</a> (2026-10-15)</h2> <h2>Key Changes</h2> <p>0.24.0 adds run-wide function error formatting and MCP discovery annotations, and improves recovery, sandbox safety, and tracing. This minor release includes compatibility changes for custom error callbacks and some saved agent graphs. Review the first two highlights before upgrading.</p> <h3>Highlights:</h3> <ul> <li><strong>Update custom error formatters.</strong> <code>RunConfig.tool_error_formatter</code> can receive <code>kind="tool_exception"</code> and an optional <code>error</code>. Handle the new kind or return <code>None</code> to retain the default message. Explicit per-tool handlers keep precedence; unconfigured defaults stay unchanged.</li> <li><strong>Plan upgrades around saved runs.</strong> New RunState snapshots use schema 1.21. Finish affected older same-name agent graphs on their original SDK. Keep the agent code and original hook ordering unchanged. Use unique agent names for new durable runs; renaming a saved graph is not a migration.</li> <li><strong>Recover completed work more reliably.</strong> Deferred approval history, completed tool outputs, aliases, and compaction state survive supported recovery paths. Automatic compaction waits for approvals. Redis replaces history atomically on supported standalone clients. Conversations appends respect the 20-item request limit. Reconcile partial writes before retrying.</li> <li><strong>Harden local execution and display.</strong> macOS sandbox commands deny Launch Services requests. Sandbox tools recover from invalid model arguments. Protected mount errors preserve caller exception data while detaching the caught traceback. The realtime demo displays tool content as text.</li> <li><strong>Honor tracing choices through resume and cleanup.</strong> Per-run tracing opt-out applies inside caller traces, and approved resumed tools attach to the resumed agent span.</li> <li><strong>Use richer integration examples and metadata.</strong> MCP tools expose optional <code>mcp_annotations</code> as discovery hints. New Decisions recipes demonstrate classification, routing, and tool review with OpenAI &gt;=3.26.0,&lt;4. They use a separate client, not an Agents model adapter.</li> <li><strong>Refresh optional dependency locks.</strong> The Temporal extra uses <code>temporalio==1.34.0</code>; the Vercel extra supports <code>vercel-sandbox&gt;=0.7.0,&lt;0.9</code>. Core Python and OpenAI SDK minimums are unchanged.</li> </ul> <p>Thanks to <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/masasron/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/masasron">@masasron</a> for reporting the macOS local sandbox issue fixed in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="5752885540" data-permission-text="Title is private" data-url="https://github.com/openai/openai-agents-python/issues/5335" data-hovercard-type="pull_request" data-hovercard-url="/openai/openai-agents-python/pull/5335/hovercard" href="https://github.com/openai/openai-agents-python/pull/5335">#5335</a>.</p> <h3>Features</h3> <ul> <li>expose MCP annotations on converted tools (<a href="https://github.com/openai/openai-agents-python/issues/5356" data-hovercard-type="pull_request" data-hovercard-url="/openai/openai-agents-python/pull/5356/hovercard">#5356</a>) (<a href="https://github.com/openai/openai-agents-python/commit/75813f83997b249e5f4b53cccaff0d8a03aa179f">75813f8</a>)</li> <li>support run-wide tool exception formatting (<a href="https://github.com/openai/openai-agents-python/issues/5326" data-hovercard-type="pull_request" data-hovercard-url="/openai/openai-agents-python/pull/5326/hovercard">#5326</a>) (<a href="https://github.com/openai/openai-agents-python/commit/8fd9a7cd19a4c495c438d39c83fb03ea3fb5a7e0">8fd9a7c</a>)</li> </ul> <h3>Bug Fixes</h3> <ul> <li>batch OpenAI conversation session writes within the API limit (<a href="https://github.com/openai/openai-agents-python/issues/5334" data-hovercard-type="pull_request" data-hovercard-url="/openai/openai-agents-python/pull/5334/hovercard">#5334</a>) (<a href="https://github.com/openai/openai-agents-python/commit/26345c1e45ebede8e2fc9b0bc7341dedab5e01fc">26345c1</a>)</li> <li>defer automatic compaction for pending approvals (<a href="https://github.com/openai/openai-agents-python/issues/5363" data-hovercard-type="pull_request" data-hovercard-url="/openai/openai-agents-python/pull/5363/hovercard">#5363</a>) (<a href="https://github.com/openai/openai-agents-python/commit/5eb47a5bdeab8ba9f6087de288c7e606816c76c6">5eb47a5</a>)</li> <li><strong>deps-dev:</strong> bump fastapi from 0.139.2 to 0.141.1 (<a href="https://github.com/openai/openai-agents-python/issues/5312" data-hovercard-type="pull_request" data-hovercard-url="/openai/openai-agents-python/pull/5312/hovercard">#5312</a>) (<a href="https://github.com/openai/openai-agents-python/commit/cdf23bd513b1fa2d26a1057784d64fdc43c8468a">cdf23bd</a>)</li> <li><strong>deps-dev:</strong> bump grpcio from 1.83.1 to 1.84.0 (<a href="https://github.com/openai/openai-agents-python/issues/5310" data-hovercard-type="pull_request" data-hovercard-url="/openai/openai-agents-python/pull/5310/hovercard">#5310</a>) (<a href="https://github.com/openai/openai-agents-python/commit/a85487ebf1b9b52cabeac6afa6c4c9110214b414">a85487e</a>)</li> <li><strong>deps-dev:</strong> bump pymdown-extensions from 11.0.2 to 12.1 (<a href="https://github.com/openai/openai-agents-python/issues/5314" data-hovercard-type="pull_request" data-hovercard-url="/openai/openai-agents-python/pull/5314/hovercard">#5314</a>) (<a href="https://github.com/openai/openai-agents-python/commit/d9758b00b53f0b0cac68955717331c943a6c2620">d9758b0</a>)</li> <li><strong>deps-dev:</strong> bump pymongo from 4.18.1 to 4.18.2 (<a href="https://github.com/openai/ope

  • 0 adds run-wide function error formatting and MCP discovery annotations, and improves recovery, sandbox safety, and tracing.
Nguyên văn nguồn
Chi tiết nguồn
  • v0.24.0 openai/openai-agents-python releases · 19 thg 11, 2026
Kỹ thuật

This featherweight, flying robot carries its own sensors and computers

<p>Over the past decade, MIT researchers have developed aerial microrobots with bumblebee-like speed, agility, and durability, for tasks like precision agriculture or search-and-rescue. But these featherweight microrobots couldn’t carry the electronics needed for navigation, forcing them to remain connected to a motion-capture system in the lab.</p><p>Now, the researchers have designed a flapping-wing microrobot with onboard sensors and computation, which can follow a complex trajectory and land precisely on a sunflower outside the lab’s motion capture environment.</p><p>While the robot is still connected to an offboard power source, this demonstration highlights the potential of aerial microrobots to be used for precision agriculture in the future.&nbsp;</p><p>For instance, flying microrobots could weave through a field of almond trees or blueberry bushes to carefully pollinate delicate blossoms in regions where natural pollinator populations have been decimated by habitat loss. They could also hover near plants without causing damage, monitoring crop health and spotting pests before they become widespread.</p><center><p class="text-align-center"><iframe width="322" height="576" src="https://www.youtube.com/embed/ZjshZHcJp30" title="A Flying Microrobot With a Built-In Brain" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen=""></iframe></p></center><p>To enable onboard sensing and computation, the researchers developed a method to expand the artificial muscles that power the robot’s wings, increasing the net payload so the tiny robot could carry the necessary electronics.</p><p>They also designed a sensor suite that can rapidly estimate the robot’s position and orientation without relying on an offboard motion capture system, and a miniature electronics package to securely hold the tiny computational components on the nimble vehicle.</p><p>“This is a really important milestone for the research community. This is the first demonstration of an aerial microrobot that is sensory and compute autonomous. There are many situations, like pollination or industrial surveillance, where you can’t use a larger robot. Small-scale robots are uniquely positioned for these types of tasks, and if this thread of technology becomes successful in the future, it could be very promising,” says Kevin Chen, an associate professor in the Department of Electrical Engineering and Computer Science (EECS), head of the Soft and Micro Robotics Laboratory within the Research Laboratory of Electronics (RLE), and co-corresponding author of a <a href="https://www.science.org/doi/10.1126/scirobotics.aeg6618" target="_blank">paper on the robot</a>.</p><p>He is joined on the paper by co-lead authors and MIT graduate students Yi-Hsuan (Nemo) Hsiao, Quang Phuc Kieu, and Zhongtao Guan; co-corresponding authors Jonathan P. How, a Ford&nbsp;Professor of Aeronautics and Astronautics and a principal investigator in the Laboratory for Information and Decision Systems (LIDS) and the Aerospace Controls Laboratory (ACL)&nbsp;at MIT; Elizabeth Farrell Helbling, an assistant professor at Cornell University; and others at MIT. The research <a href="https://www.science.org/doi/10.1126/scirobotics.aeg6618" target="_blank">appears today in <em>Science Robotics</em></a>.</p><p><strong>Increasing the payload</strong></p><p>The MIT researchers previously developed a&nbsp;<a href="https://news.mit.edu/2025/mit-engineers-design-aerial-microrobot-fly-like-bumblebee-1203" target="_blank">swift, agile aerial microrobot</a> with four wings, each powered by a set of soft actuators, or artificial muscles, that flap the wings rapidly so the bot can perform detailed maneuvers.&nbsp;</p><p>The robot, roughly the same size and shape as a microcassette, weighs less than a paperclip.</p><p>To bring this tiny robot closer to real-world applications, the researchers need it to carry the additional weight from electronics for onboard sensing and computation, while still being fast, maneuverable, and robust to collisions.</p><p>In this research, they developed a method to scale up the artificial muscles without hampering the robot’s performance. They carefully tuned the radius and length of the soft actuators, increasing their size by 50 percent while boosting the robot’s net lift forces by 46 percent.&nbsp;</p><p>“It is very difficult to scale up the actuators without causing the output power of the system to go down, which reduces the lift forces. We tuned the geometry of the robot very precisely so lift proportionally increases with the additional actuator weight,” Chen explains.&nbsp;</p><p>They were able to achieve a lift-to-weight ratio of 1.9, giving the robot more than enough lift force to carry the onboard electronics package they developed, which weighs only 244 milligrams.&nbsp;</p><p>The tiny electronics package contains the sensors, and microcontroller needed for onboard sensing and computation.&nbsp;</p><p>The researchers incorporated an inertial measurement unit to estimate attitude, an optical flow sensor to estimate lateral flight speed and position, and a time-of-flight sensor to measure height.</p><p>But without the precision of the attached motion capture system, the robot does not know its absolute position, causing it to drift off course.</p><p><strong>Speedy sensing</strong></p><p>To overcome this challenge, the team designed a sensing algorithm to calculate the robot orientation, angular velocity, and position extremely quickly, simplifying the calculations to reduce the computational cost.</p><p>They found that estimating position continuously at a faster rate yielded higher performance than a more accurate, slower estimation.</p><p>“We spent a lot of time optimizing the selection of the sensors and position estimators so the robot would have very low drift. Our robot can now fly with positional accur

  • &nbsp;</p><p>“It is very difficult to scale up the actuators without causing the output power of the system to go down, which reduces the lift forces.
Nguyên văn nguồn
Chi tiết nguồn
Kiến trúc

How Stretto built an AI voice agent with Amazon Nova 2 Sonic

<p><em>This post is co-written with Danya Gordin from Stretto and Marcelo Wzorek from CI&amp;T.</em></p> <p>Before <a href="https://www.stretto.com/" target="_blank" rel="noopener">Stretto</a> built an AI voice agent, consumers considering bankruptcy on its Bankruptcy.com site filled out a web form with a few fields: debt amount, attorney status, and contact information. These form-generated leads carried thin margins.</p> <p>The gap between a web form lead and a fully qualified lead is significant. Bankruptcy attorneys place more value on a lead that includes detailed financial data: debts, assets, vehicles, real estate, employment history, and prior filings. The web form could have been extended to capture that detail, but the longer it got, the more visitors abandoned it partway through. A phone interview would have drawn the information out naturally, because people are more willing to share financially sensitive details in conversation, but staffing and training a human intake team to run those interviews was cost-prohibitive.</p> <p>Stretto saw an opportunity: replace the web form with an AI voice agent that conducts the intake interview by phone. The agent would guide callers through the full bankruptcy questionnaire conversationally and extract structured data in real time. It would then sort leads against attorney-defined rules to support attorney decision-making.</p> <p>The challenge was finding a voice AI solution that could handle sensitive financial conversations naturally, with low latency, clean interruption handling, and emotional awareness, at a cost that made the unit economics work.</p> <p>In this post, we walk through how Stretto worked with the <a href="https://aws.amazon.com/ai/generative-ai/innovation-center/" target="_blank" rel="noopener">AWS Generative AI Innovation Center</a> and <a href="https://ciandt.com/" target="_blank" rel="noopener">CI&amp;T</a> (an AWS Partner) to build Tara, an AI voice agent powered by <a href="https://docs.aws.amazon.com/nova/latest/nova2-userguide/using-conversational-speech.html" target="_blank" rel="noopener">Amazon Nova 2 Sonic</a>, a speech-to-speech model in Amazon Bedrock. Tara conducts multi-phase bankruptcy intake interviews and sorts leads against attorney-defined rules. We cover the architecture, the phased build from validation through production deployment, and how the team tested voice quality for sensitive conversations.</p> <h2 id="solution-overview">Solution overview</h2> <p>Tara is an AI voice agent that conducts bankruptcy intake interviews over the phone. When a caller dials the Bankruptcy.com number, Tara picks up, introduces herself, and guides the caller through a structured conversation. She collects personal information, financial details, asset inventories, and employment history while adapting her tone to the caller’s emotional state. At the end of the call, Stretto’s system applies attorney-defined rules to sort the lead as qualified, disqualified, or needs-more-information for attorney review.</p> <p>The solution is built on Amazon Nova 2 Sonic, a speech-to-speech foundation model (FM)available through Amazon Bedrock. It processes speech input and generates speech output natively in a single model rather than chaining separate speech-to-text, language model, and text-to-speech components. This architecture is designed to reduce latency and produce more natural-sounding conversations, both critical for the sensitive nature of bankruptcy intake calls. Building on Amazon Bedrock gave the team both Amazon Nova 2 Sonic and Anthropic’s Claude through one managed service, with no model infrastructure to run. It also brought AWS security and governance controls to a workload that handles sensitive financial data. Access to both models is governed by AWS Identity and Access Management (IAM). Traffic to Amazon Bedrock stays on the AWS network through virtual private cloud (VPC) endpoints (AWS PrivateLink). Amazon Bedrock doesn’t use prompts or model outputs to train models (for details, see <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/data-protection.html" target="_blank" rel="noopener">Data protection in Amazon Bedrock</a>). Because the models run in the same AWS environment as the voice bridge, the database, and Amazon CloudWatch monitoring, the team could secure and operate the whole solution with one set of tools. Amazon Nova 2 Sonic and Amazon Bedrock are not available in all AWS Regions. For current Region availability, see <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/models-regions.html" target="_blank" rel="noopener">Amazon Bedrock supported Regions</a>.</p> <p>The team chose Amazon Nova 2 Sonic over the previous voice solution for three reasons:</p> <ul> <li><strong>Cost:</strong> Stretto reports a 95 percent reduction in per-minute voice cost compared to its previous solution, based on <a href="https://aws.amazon.com/bedrock/pricing/" target="_blank" rel="noopener">Amazon Bedrock on-demand pricing</a> in us-east-1 as of April 2026.</li> <li><strong>Latency:</strong> In Stretto’s testing, the previous solution exhibited time-to-first-audio latency above 500 milliseconds, creating awkward pauses. Amazon Nova 2 Sonic is designed for low latency to support natural conversational flow.</li> <li><strong>Interruption handling:</strong> Callers in stressful situations need to interject, correct themselves, and ask clarifying questions. The previous solution handled barge-in poorly, often breaking the conversation flow. Amazon Nova 2 Sonic is designed to handle interruptions without breaking the session.</li> </ul> <p>The following figure shows the end-to-end architecture.</p> <div style="width: 810px" class="wp-caption alignnone"> <a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/16/ML-21890-1.jpg" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/16/ML-2189

  • This architecture is designed to reduce latency and produce more natural-sounding conversations, both critical for the sensitive nature of bankruptcy intake calls.
Nguyên văn nguồn
Chi tiết nguồn
Tác tử

Quoting Daniel Stenberg

<blockquote cite="https://daniel.haxx.se/blog/2026/11/17/actual-curl-rewrites-in-rust/"><p>During the time these agents converted 185,000 lines of C code into 340,000 lines of Rust code (I counted), we merged over 300 commits in the actual curl project. We actually improved curl during these a little more than three weeks. A little over 200 bugfixes actually. 200 bugfixes that none of those rust rewrites have. We also fixed a few vulnerabilities. In another month we have done another 200 and maybe fixed a few more vulnerabilities. Will these clones keep up?</p></blockquote> <p class="cite">&mdash; <a href="https://daniel.haxx.se/blog/2026/11/17/actual-curl-rewrites-in-rust/">Daniel Stenberg</a>, Actual curl rewrites in Rust</p> <p>Tags: <a href="https://simonwillison.net/tags/daniel-stenberg">daniel-stenberg</a>, <a href="https://simonwillison.net/tags/curl">curl</a>, <a href="https://simonwillison.net/tags/rust">rust</a>, <a href="https://simonwillison.net/tags/security">security</a>, <a href="https://simonwillison.net/tags/coding-agents">coding-agents</a>, <a href="https://simonwillison.net/tags/c">c</a>, <a href="https://simonwillison.net/tags/ai">ai</a>, <a href="https://simonwillison.net/tags/llms">llms</a>, <a href="https://simonwillison.net/tags/generative-ai">generative-ai</a>, <a href="https://simonwillison.net/tags/vibe-porting">vibe-porting</a></p>

  • We actually improved curl during these a little more than three weeks.
Nguyên văn nguồn
Chi tiết nguồn
Đánh giá

[AINews] GPT-6.2 Astra, the #2 Frontier Model is better and 20%-600% cheaper, but has mixed reviews; 6.2 Sol likely main workhorse

OpenAI shipped GPT-6.2 Astra and Sol today, two days before Anthropic’s expected first trading day . In Artificial Analysis ’ tests, Sol ties September’s Astra at about 15% of the cost per task . New Astra scores 62 on the Intelligence Index, just behind Fable 5.2 at 65. Net net, Anthropic still owns the world’s best model, but OpenAI owns the world’s most efficient models in terms of intelligence per $. What’s more unusual about this launch is the STRONGLY negative influencer feedback… which prompted a defensive response … View post on X ↗ We think that some of this is overreacting and normal influencer tendency to either have everything be 🤯🤯🤯 or the literal WORST.MODEL.EVER, BUT also that some of the criticisms are valid, and the RL artefacts are noticeable and downright bizarre (don’t even get us started about the terrible launch video ). However, we still use it to automate our work and that’s all that matters in the end. Great ship, but Anthropic’s IPO is unaffected and that will be probably the headline of the week/year/decade (until OpenAI IPO’s….) Today’s AI News is brought to you by SerpApi! Retrieve public web pages by URL as LLM-ready Markdown or structured JSON. Map sites and run bounded crawls with one API. Pair with 100+ SerpApi search APIs for search-to-content workflows. Check them out to support LS, and contact business@latent.space for feedback! AI Twitter Recap GPT-6.2 Astra and Sol Availability. Today’s models are rolling out to ChatGPT Work and Codex for Plus, Pro, Enterprise and Edu, and to developers through the API as gpt-6.2-astra and gpt-6.2-sol . OpenAI API prices per million tokens. Astra is $8 input, $0.40 cached input and $40 output. Sol keeps its predecessor’s pricing at $2, $0.10 and $10. Both have 1M context and a 128K output limit. Vals · Ultrafast pricing and rollout Astra for Law. The Enterprise configuration adds a legal search index. Artificial Analysis measured 20.8% on Harvey LAB-AA, versus 15.0% for base Astra. Artificial Analysis Preparedness. OpenAI rated Astra High for AI Self-Improvement, a first for one of its externally released models. On a separate measure, Vals’ RSI-Index rose from 27.07% to 29.01%, with 50% denoting its human reference. System card · Vals FrontierMath Erdős. Epoch verified seven solutions in Lean out of 68 problems. The previous best was three. Epoch What the benchmarks measure Agents’ Last Exam 1.1. OpenAI reports Astra at xhigh gained 5.2 points over GPT-6 Astra at max, using less than a third of the cost per task. It reports a 5.5-point lead over Fable 5.2 at about 5% of the cost on that test. On GDPval v2, it says Astra used 22% fewer output tokens than its predecessor. OpenAI’s results Long engineering runs. Mercor’s OpenWorldEngineer puts Astra at 67.2%, Fable 5.2 at 54.6%, and GPT-6 Astra at 44.2%. On Vals’ multi-turn Vibe Code Bench, Astra was first and Sol was third. Mercor · Vals 3D and robotics. Astra led AA-3D with 1,318, ahead of Fable at 1,271 and Sol at 1,259. On Artificial Analysis’s robotics preview, Astra scored 79 and Sol 78, ahead of Fable at 75. Artificial Analysis Research agents. On Perplexity’s WANDR, Astra scored 0.783 at about $12.54 per task. Sol reached 0.746 at about $2.65. Perplexity Early testers disagreed Matt Shumer called the models “essentially unusable” after weeks of testing and said his Opus and Fable agents rejected much of Astra’s subagent work. Review · Workflow details Pietro Schirano attributed many of his early issues to the harness. Arena’s Peter Gostev tested Astra against Fable on visual coding and singled out Astra’s SVG peacock. Schirano · Gostev The demos. Gabriel Chua built a walkable Singapore using more than 100 Codex threads. Simon Willison ran the same three.js mech prompt through several models, including Astra, Sol, Carbon and Fable. Singapore build · Same-prompt comparison Microsoft Ignite and Gemini Microsoft ships its own coding models. MAI Code 2 is for multi-step coding, and Code 2 Flash is for quicker edits. Both are available in GitHub Copilot’s model picker, Copilot Fusion and Foundry. Microsoft also announced a multimodal upgrade to its Decision model. Microsoft Copilot Fusion. Microsoft reports 71.2% on Terminal-Bench 4.1 with its multi-model harness, at 88% lower cost than Claude Fable 5.2 in that comparison. Jay Parikh Image editing. Artificial Analysis ranks MAI Image 2.7 first on its editing benchmark, at $34.82 per 1,000 images. Artificial Analysis Gemini Tasks. Hydrogen now powers free Tasks in six countries, including tasks that need a phone call or a text. Google also showed a new Antigravity robotics plugin with MuJoCo simulation. Sundar Pichai · Google DeepMind Agent harnesses and inference Hugging Face’s Pi adapter. A small adapter trained inside pi lifted Qwen3.8-27B from 15 to 32 solves on the 51 hardest SWE-bench Pro tasks. Across that subset it made 55% fewer tool calls per task. Model card and paired results Harness costs vary sharply. Baseten measured up to a 4.4× spread in cost per passing task across its benchmark of 22 harnesses and five open models. Baseten The harness may specialize to its model. Google’s SIM keeps the rationale and outcome of prior harness edits. A harness it evolved on Opus scored 68.41 on Gemini, versus 73.89 for the starting harness. Research summary More multi-agent products. Cognition launched persistent Devin Bots. Perplexity reported that its mixed-model teams beat four instances of the lead model on Terminal-Bench and WANDR. Replit reported diminishing returns from 16 to 32 agents on NumPy tasks. Cognition · Perplexity · Replit Coordination dominated a 1,000-agent run. Hugging Face says only 40–60 agents were calling the model at any moment. They read roughly 130 tokens for every token they wrote, with merges and tests contributing to the bottleneck. Thomas Wolf A small fraction of cache misses can dominate the bill. SemiAnalysis reports that 1.8% of uncached requests accounted for roughly 15% of spend i

  • OpenAI rated Astra High for AI Self-Improvement, a first for one of its externally released models.
  • OpenAI’s results Long engineering runs.
  • Model card and paired results Harness costs vary sharply.
Nguyên văn nguồn
Chi tiết nguồn
Kỹ thuật

Ask HN: Can world models lead to racisim? Are there any Alignment frameworks?

Câu hỏi I am not sure how the alignment of World Models will happen?

I have never felt or believed in Dooms day theories about AI. But below is one of only 2 occasions where I am feeling AI can be too dangerous. While building world model for enterprise decisions, I am intrigued by how far the AI can go out of reach. Got to know - that now world models can learn from emotions & genetics. That means - actually predicting the outcome & behavior of humans. If going unabated - so the builder can bring in bias. It sounds Deja vu for when the racist theories developed in wraps of the scholarly research in 1900s. When I understand how the world model is gonna work - practically emotions & genetics aren't very far away from being built. There are already startups working on it. While ours is only for Enterprise decisions, the one on genetics as world-wide population scale impact. For World Models for Physical AI or Humanoids - this is more of physical tasks. I don't call it dangerous as there are quite a few checks & controls. But when world models are used for Genetics - this has far reaching unabated repercussions on how humans bheave with each other. As models become more powerful, ppl will start relying on it. I am not sure how the alignment of World Models will happen? LLMs are tbh primitive in nature. For World Models alignment - are there any specific efforts or initiatives wroking on it? We would be glad to include any alignment frameworks in our Benchmarking for Enterprise decisions. To monitor & remove bias.

HN: AI agent · atul-mehra 5 lượt bình chọn 0 bình luận Mở bản gốc ↗
Nguyên văn nguồn
Chi tiết nguồn
Tác tử

Research acceleration: The jagged frontier

OpenAI measures how coding agents are accelerating AI research, where their performance remains uneven, and what this signals about recursive self-improvement.

  • OpenAI measures how coding agents are accelerating AI research, where their performance remains uneven, and what this signals about recursive self-improvement.
Nguyên văn nguồn
Chi tiết nguồn
Kỹ thuật

Advancing chip design with OpenAI models

OpenAI models optimized circuits used in Jalapeño silicon and improved RISC‑V core designs, showing progress in AI-assisted chip design with GPT‑6 Astra.

  • OpenAI models optimized circuits used in Jalapeño silicon and improved RISC‑V core designs, showing progress in AI-assisted chip design with GPT‑6 Astra.
Nguyên văn nguồn
Chi tiết nguồn
Đánh giá

Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference

<p>Serving large language models (LLMs) efficiently requires sufficient GPU memory and careful coordination of prompt processing and token generation. GLM-5.3’s FP8 weights, for example, exceed the 640 GB of GPU memory on a single <code>ml.p5.48xlarge</code> instance, before accounting for KV cache and runtime overhead. Long prompts can also slow token generation when both share the same GPUs.</p> <p>Amazon SageMaker HyperPod Inference now supports <a href="https://github.com/ai-dynamo/dynamo" target="_blank" rel="noopener">NVIDIA Dynamo</a>, combining Dynamo’s distributed inference capabilities with the infrastructure management of HyperPod to address these challenges. Dynamo is an open source distributed inference framework that supports multi-node deployment, disaggregated prefill and decode, and KV-aware routing with engines such as vLLM, SGLang, and TensorRT-LLM. HyperPod supports these deployments with high-performance Elastic Fabric Adapter (EFA) networking, coordinated GPU scheduling, and node health monitoring and recovery. In this post, we illustrate multi-node and disaggregated serving through a deployment example, share benchmark results, and outline best practices for optimizing inference on HyperPod.</p> <h2 id="how-the-integration-works">How the integration works</h2> <p>The HyperPod Inference add-on installs the Dynamo custom resource definitions and these serving and scheduling components:</p> <ul> <li><em>HyperPod Inference Operator </em>manages endpoint deployment through an InferenceEndpointConfig (IEC) custom resource. Its <code>dynamoSpec</code> section supplies the Dynamo configuration.</li> <li><em>Dynamo operator </em>manages resources defined in a <a href="https://docs.nvidia.com/dynamo/dev/kubernetes/model-deployment/deploy-with-dgd" target="_blank" rel="noopener">DynamoGraphDeployment (DGD)</a>, including the frontend, model workers, replica counts, and resource requirements.</li> <li><em><a href="https://github.com/ai-dynamo/grove" target="_blank" rel="noopener">Grove</a> </em>coordinates related groups of pods, including multi-node workers, and their scaling relationships.</li> <li><em><a href="https://github.com/kai-scheduler/KAI-Scheduler" target="_blank" rel="noopener">Kubernetes AI (KAI) Scheduler</a> </em>places pods on GPU resources, using gang scheduling to schedule the pods required for a multi-node worker together.</li> </ul> <p>AWS Deep Learning Containers (DLCs) provide the Dynamo runtime, while the add-on manages integrated component versions. To adopt a supported Dynamo release, update the add-on.</p> <h3 id="deployment-architecture">Deployment architecture</h3> <p>The following diagram shows the deployment paths, serving components, and infrastructure management for disaggregated inference on SageMaker HyperPod. The Dynamo frontend routes requests to prefill workers, which process prompts, and decode workers, which generate tokens. These separate pools allow independent scaling and tuning. Both pools load the full model weights, and KV cache transfers between them over EFA.</p> <p>Dynamo also supports aggregated serving, where prefill and decode run in the same worker. In either topology, workers can run on one instance or span multiple instances using tensor and pipeline parallelism. Dynamo’s <a href="https://docs.nvidia.com/dynamo/dev/knowledge-base/concepts/system-architecture/kv-aware-routing" target="_blank" rel="noopener">KV-aware router</a> considers reusable prefix KV cache and active worker load in both topologies. Prefix reuse can reduce prompt recomputation and improve time to first token, while load awareness helps avoid overloading workers. For more details on the Dynamo architecture, see the <a href="https://docs.nvidia.com/dynamo/knowledge-base/overview" target="_blank" rel="noopener">Dynamo Knowledge Base overview</a>.</p> <p>HyperPod provides node, GPU, and EFA health monitoring and manages node replacement, while the orchestration components recreate affected workers. Deployments configured through InferenceEndpointConfig with <code>dynamoSpec</code> support invocation through SageMaker endpoints.</p> <div style="width: 748px" class="wp-caption alignnone"> <a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/14/ML-22230-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/14/ML-22230-1.png" alt="Architecture diagram of NVIDIA Dynamo on SageMaker HyperPod with the frontend routing to prefill and decode workers over EFA" width="738"></a> <p class="wp-caption-text">Figure 1: Deployment paths, serving components, and infrastructure management for NVIDIA Dynamo on Amazon SageMaker HyperPod Inference</p> </div> <h2 id="two-ways-to-configure-your-deployment">Two ways to configure your deployment</h2> <p>Whether you are new to HyperPod, new to Dynamo, or both, the add-on offers two deployment paths to fit your workflow:</p> <p><em>Bring-your-own DynamoGraphDeployment (DGD)</em> lets existing Dynamo users continue using upstream documentation, recipes, and manifests, with the Dynamo operator managing the deployment. To bring an existing DGD to HyperPod, use a supported container image and configure storage, networking, and pod placement for your cluster.</p> <p><em>InferenceEndpointConfig with dynamoSpec</em> lets existing HyperPod Inference users deploy a new Dynamo endpoint using the same configuration format and OpenAI-compatible interface they already know. For the inference engine, you can choose vLLM, SGLang, or TensorRT-LLM, with vLLM used by default. The operator also supplies defaults for KV-aware routing and replica counts.</p> <p>Both paths support single-instance and multi-node deployments. In either case, HyperPod manages node health and replacement. The table summarizes the responsibilities for each path.</p> <table border="1px" width="100%" cellpadding="10px"> <tbody> <tr> <td></td> <td><strong>Bri

  • Prefix reuse can reduce prompt recomputation and improve time to first token, while load awareness helps avoid overloading workers.
Nguyên văn nguồn
Chi tiết nguồn
Đánh giá

Reducing token costs and latency in Amazon Bedrock AgentCore JSON workloads

<p>As enterprises build production agents with dozens of tools and large structured inputs, token consumption and latency become the primary blockers to a good user experience. When these agents process large JSON data, every additional tool invocation injects the full payload into the large language model (LLM) context window, causing token usage and response latency to spike. Over multi-turn conversations, this pressure compounds, driving up costs and degrading response quality. Effectively managing context-window pressure in agentic systems is key to reliability and responsiveness at scale. In this post, we share three composable strategies we used to build a production agent for Flightscape that handles 20 MB+ JSON payloads. You can apply them to reduce token costs, lower latency, and maintain full reasoning capability across extended sessions. Together, these optimizations reduced token consumption by up to 71% and cut average latency by up to 80% across our benchmark runs on a representative 23 MB payload.</p> <h2 id="solution-overview">Solution overview</h2> <p>Flightscape is an airline operations platform that helps airlines plan and manage their crews, flights, and operations. As part of Flightscape’s crew management capabilities, we built a Training Schedule Optimization agent that verifies crew training schedule assignments against business rules. With 40+ tools and JSON inputs in megabytes, this agent could answer complex queries across crew assignments, scheduling, and resource allocation. Although the agent delivered the required functionality, average request latencies exceeded 15 seconds across most calls. Every tool call injected its full payload into the model context, inflating token counts and costs. On larger datasets, the context window filled up before the agent could finish reasoning. The following diagram (Figure 1) details the agentic architecture and data flow:</p> <div style="width: 810px" class="wp-caption alignnone"> <a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/18/ml-21786-Figure-1-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/18/ml-21786-Figure-1-1.png" alt="Architecture diagram: the optimization workflow outputs request, response, and validation report JSON, which an AI agent reads to answer user questions through a chat interface" width="800"></a> <p class="wp-caption-text">Figure 1: Agent architecture and data flow</p> </div> <p>In Figure 1, the Optimizer receives a Request JSON and produces a Response JSON, which is then validated by a Business Rules Validator to generate a Validation Report JSON. All three JSON documents are ingested by an AI Agent that provides a conversational interface for users to query optimization results, understand validation errors, and explore scheduling assignments. The agent is built with <a href="https://www.langchain.com/langgraph" target="_blank" rel="noopener">LangGraph</a>, an open source framework by LangChain, and runs on <a href="https://aws.amazon.com/bedrock/agentcore/" target="_blank" rel="noopener">Amazon Bedrock AgentCore</a>, a platform to build, connect, and optimize agents at scale, with any framework or model. Amazon <a href="https://aws.amazon.com/bedrock/guardrails/" target="_blank" rel="noopener">Bedrock Guardrails</a> can be applied to validate and filter the agent’s conversational responses, helping facilitate responsible AI practices in production deployments. We defined a custom agent state that holds the data needed by our tools: optimization request JSON, optimization response JSON, and validation report JSON. We also added basic statistics and cache data to state for rich queries. The agent uses an <a href="https://aws.amazon.com/bedrock/" target="_blank" rel="noopener">Amazon Bedrock</a> model for reasoning and <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory.html" target="_blank" rel="noopener">Amazon Bedrock AgentCore memory</a>, a capability of Amazon Bedrock AgentCore, to persist chat interactions across turns, and each turn follows this flow (Figure 2):</p> <div style="width: 810px" class="wp-caption alignnone"> <a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/13/ML-21786-2.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/13/ML-21786-2.png" alt="Diagram of one agent turn: hydrate session state, run tool calls, then dehydrate state back to storage" width="800"></a> <p class="wp-caption-text">Figure 2: Agentic flow</p> </div> <p>This is what the state structure looks like. The state references TOPT (Training Optimizer) request and response objects, which represent the crew scheduling inputs and optimization results:</p> <div class="hide-language"> <pre><code class="language-python">class GraphState(TypedDict): """LangGraph state that combines messages with application data.""" # Messages from conversation (managed by add_messages reducer) messages: Annotated[List[BaseMessage], add_messages] # Application-specific state - parsed domain objects request: Optional[TOPTRequest] responses: List[TOPTResponse] validation_report: Optional[Dict[str, Any]] basic_stats: Optional[Dict[str, Any]] cache_data: Optional[Dict[str, Any]]</code></pre> </div> <p><strong>Note:</strong> This state originally held heavy payloads inline. Later sections explain how we moved large artifacts out of conversational context to reduce token usage.</p> <p>We implemented three strategies to reduce the token costs, latency, and memory overhead caused by passing large JSON payloads through the model context. The following diagram (Figure 3) shows how the three strategies fit into a single agent turn. Strategy 1 runs at the start and end of each turn, and Strategies 2 and 3 run each time a tool returns a result.</p> <div style="width:

  • You can apply them to reduce token costs, lower latency, and maintain full reasoning capability across extended sessions.
  • All three JSON documents are ingested by an AI Agent that provides a conversational interface for users to query optimization results, understand validation errors, and explore scheduling assignments.
  • """ # Messages from conversation (managed by add_messages reducer) messages: Annotated[List[BaseMessage], add_messages] # Application-specific state - parsed domain objects request: Optional[TOPTRequest] responses: List[TOPTResponse] validation_report: Optional[Dict[str, Any]] basic_stats: Optional[Dict[str, Any]] cache_data: Optional[Dict[str, Any]]</code></pre> </div> <p><strong>Note:</strong> This state originally held heavy payloads inline.
Nguyên văn nguồn
Chi tiết nguồn
Huấn luyện

Amazon SageMaker AI real-time endpoints now support KV cache aware routing

<p>Today, Amazon SageMaker AI introduces KV cache-aware routing (KV-aware routing), a new routing strategy for real-time endpoints. It uses the cache state that the model container reports. It sends each request to an instance that already holds a long cached match for its prompt and isn’t too busy. It’s for teams that serve large language models (LLMs) with multi-turn conversations, long shared prompts, or heavy load. You turn it on in two places: add one environment variable to your model so the container reports its cache contents, and set <code>RoutingStrategy</code> to <code>KV_AWARE</code> in the endpoint configuration. Your clients call the endpoint with the same APIs as before.</p> <p>Here’s the problem it solves: LLM applications such as chat assistants, Retrieval Augmented Generation (RAG), and agents with long system prompts send many requests that start with the same text. A model keeps the work it did on a prompt in GPU memory, the KV cache, and skips that work when the same start reaches the same instance again. In <a href="https://aws.amazon.com/blogs/machine-learning/reduce-llm-latency-with-prefix-aware-routing-on-amazon-sagemaker-inference/" target="_blank" rel="noopener">part 1 of this series</a>, we introduced prefix-aware routing for Amazon SageMaker AI, which sends requests that start the same way to the same instance. Prefix-aware routing infers where a prefix is cached from the request alone, so it can’t tell when an instance has evicted that cache.</p> <p>Prefix-aware routing picks an instance from the request alone. It hashes the start of the request, up to the <code>PrefixLength</code> you configure (bytes of the <code>InvokeEndpoint</code> payload, or characters of the message text for the OpenAI-compatible Chat Completions API), and sends requests that start the same way to the same instance. It never sees what an instance actually has cached. This works well when every request starts with the same shared text, and less well in two cases:</p> <p>The cache on an instance changes over time. An instance evicts older KV cache entries when its GPU memory fills up, but the router keeps sending matching requests there as if the cache were still warm.</p> <p>A conversation starts out short. In a multi-turn chat, each turn adds to the history. While the request is still shorter than <code>PrefixLength</code>, every new turn changes the text the router hashes, so the next turn can land on a different instance than the one that cached the earlier turns. The turns of a conversation stay on one instance only after the request grows past <code>PrefixLength</code>.</p> <p>In this post, we explain how KV-aware routing works, when to use it, and how to enable, invoke, and verify it on single model endpoints and inference component endpoints.</p> <h2 id="how-kv-aware-routing-works">How KV-aware routing works</h2> <p>KV-aware routing has three parts: the model container reports what it has cached, SageMaker AI keeps a managed cache index of that state for the endpoint, and the endpoint uses the index to choose an instance for each request.</p> <p>When you turn events on with <code>SM_VLLM_KV_EVENTS_CONFIG</code>, the container publishes a KV cache event each time it stores or evicts cache entries, and SageMaker AI uses these events to keep the index current. The index holds only pointers: one-way, truncated hashes of cached content, each mapped to the instance or inference component copy that holds it. It never stores prompts, token IDs, request payloads, or KV tensors. When a request arrives, the endpoint tokenizes the prompt and looks up the tokens in the index. The index returns the instances or inference component copies that hold part of the prompt in their cache, and how much of it each one holds. For example, if instance A holds the cached start of a conversation, the next turn of that conversation goes to A. Because the index matches on tokens rather than a fixed number of bytes, a multi-turn history can match in full as it grows. A request that shares only part of its prompt still goes to the instance with the longest match.</p> <p>The endpoint then picks among these candidates by weighing how much of the prompt each one has cached against how busy it is. A candidate that is too busy is skipped, so if the instance with the longest match is busy, the request goes to the next-best one that is available. If no candidate has a useful match or all of them are busy, the request falls back to load-aware routing, as described in Routing overhead and fallbacks. You don’t configure a concurrency threshold for this, as you do with prefix-aware routing. The index also follows the fleet. When you scale in, entries for removed instances are dropped. When you scale out or an update replaces instances, the new instances start with empty caches, so latency can be higher for a short time.</p> <div style="width: 810px" class="wp-caption alignnone"> <a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/17/ML-22381-1-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/17/ML-22381-1-1.png" alt="KV-aware routing: the router uses the managed cache index to send each request to the instance holding the longest cached match" width="800"></a> <p class="wp-caption-text">Figure 1: KV-aware routing on a SageMaker AI endpoint. The router looks up the managed cache index and forwards the request to the instance that holds the longest cached match. Each instance reports KV cache events back to the index</p> </div> <p>In Figure 1, the endpoint has three instances, A, B, and C. Each instance runs the same model in its own model container (the AWS vLLM Deep Learning Container listed in Requirements and limits) and has its own KV cache in GPU memory, so each instance holds different cached content: A holds turns 1 to 3 of conversation 1, B holds conversations 2 and 3, and C holds doc

  • com/blogs/machine-learning/reduce-llm-latency-with-prefix-aware-routing-on-amazon-sagemaker-inference/" target="_blank" rel="noopener">part 1 of this series</a>, we introduced prefix-aware routing for Amazon SageMaker AI, which sends requests that start the same way to the same instance.
Nguyên văn nguồn
Chi tiết nguồn
Huấn luyện

Govern and discover records with common metadata in AWS Agent Registry

<p>As organizations build more agents, Model Context Protocol (MCP) servers and skills, they need a consistent way to describe, govern, and find them. <a href="https://aws.amazon.com/blogs/machine-learning/manage-agents-tools-and-skills-at-scale-with-aws-agent-registry/" target="_blank" rel="noopener">AWS Agent Registry</a> provides a catalog for registering, approving, and discovering these resources and custom records. Each record carries a name, a description, a protocol descriptor, and an approval status. However, every organization has attributes and facts that it wants to track, and some are unique to that organization, such as its business units, resource owners, risk classifications, and development environments. Organizations want one common definition for each of these attributes, and they want every agent, MCP server, and skill in the registry to carry these facts. Until now, organizations had no way to define these facts in AWS Agent Registry and store them on each record.</p> <p>Custom metadata schemas are now available in AWS Agent Registry. A custom metadata schema is a set of organization-defined key-value pairs that records in an AWS Agent Registry carry. An administrator defines the schema and applies it to the registry, and all records within the registry must then adhere to the schema. When publishers create or submit records, they supply values for the fields specified in the applicable schema, and AWS Agent Registry validates those values against the schema set by the admin. Approvers can rely on the validated values when reviewing a record, and consumers and agents can use them to discover and filter records. Administrators, publishers, security and governance teams, approvers, consumers, and agents all work from a shared set of facts to govern, discover, and understand the resources in their agentic fleet.</p> <p>In this post, we follow the AnyCompany organization publishing an MCP server with custom schema definition, enriching that record, and making it available for discovery by humans or agents. We use an MCP server as the example, but the same steps apply to other record types.</p> <h2 id="how-custom-metadata-works">How custom metadata works</h2> <p>With custom metadata, an organization can use standardized vocabulary for describing its resources. The registry administrator defines the <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/registry-custom-metadata-schema.html" target="_blank" rel="noopener">custom metadata schema</a>: field names, types, allowed values, and whether each field is required or optional. The registry enforces the schema on every record, so a field such as <code>environment</code> has the same name and the same allowed values whichever team publishes the record.</p> <p>The schema has common attributes that apply to every record type, and you can add dedicated fields that only apply to specific record types. Each field is Text, Enum, URL, or Boolean.</p> <ul> <li> <p>When a record is created or updated, the registry validates the custom metadata against the schema set up by the admin.</p> <ul> <li> <p>If a record is created <strong><em>without any custom metadata,</em></strong> AWS Agent Registry saves it as <code>NON_COMPLIANT</code>. A team can save work in progress this way, but it can’t submit the record for approval until it has the required fields and values.</p> </li> <li> <p>If a record <strong><em>includes custom metadata</em></strong> with an unknown field, a missing required field, or an invalid value, AWS Agent Registry rejects it with a <code>ValidationException</code>.</p> </li> </ul> </li> <li> <p><code>GetRegistryRecord</code> returns the record’s custom metadata values and its compliance status. (<code>COMPLIANT</code> means that the values match the administrator’s schema. It is not an approval. The approvers make that decision.)</p> </li> <li> <p>Discovery APIs also allow filtering records based on custom attributes.</p> </li> </ul> <h2 id="what-custom-metadata-makes-possible">What custom metadata makes possible</h2> <p>Because teams can describe the AWS Agent Registry records with the same fields and values, the registry can support and scale governance and compliance.</p> <h3 id="ownership-and-operations">Ownership and operations</h3> <p>Fields such as <code>environment</code> (<code>prod</code>, <code>staging</code>, or <code>dev</code>), <code>businessUnit</code>, <code>ownerTeam</code>, <code>supportContact</code>, and <code>costCenter</code> record where a resource runs, who is responsible for it, and who pays for it. A configuration management database (CMDB) ID links the record to your existing inventory, and a support-tier field sets expectations for anyone who depends on the resource.</p> <h3 id="security-and-compliance">Security and compliance</h3> <p>Data classification, compliance scope such as Payment Card Industry (PCI) or Health Insurance Portability and Accountability Act (HIPAA), risk tier, scan status, and a scan report URL turn an approval into a check of structured evidence. Your existing scanners, CMDB sync jobs, and classification tools write these values through the API before a record is submitted. You can build your record approval process or automation using these values, as this post shows.</p> <h3 id="better-discovery">Better discovery</h3> <p>Custom metadata values are indexed for search. A consumer or agent can combine a natural-language query with specific metadata filters, such as <code>environment=prod AND businessUnit=payments AND securityScanStatus=PASSED</code>. Only approved records are returned, so a record that matches the filters but hasn’t been approved stays out of the results. <em>Figure 1</em> shows how each role, from administrator to consumer, moves a record through schema definition, registration, enrichment, approval, and discovery.</p> <div style="width: 810px" class="wp-caption alignnone"> <a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea

  • Only approved records are returned, so a record that matches the filters but hasn’t been approved stays out of the results.
Nguyên văn nguồn
Chi tiết nguồn
Đánh giá

Announcing AI data preparation capabilities in Amazon SageMaker: Build high-quality datasets for model development workflows

<p>Teams that ship generative AI to production know that a model is only as good as its data. That is why preparing high-quality training and evaluation data comes first. Fine-tuning an open-weight model, running reinforcement learning, or building an evaluation suite all depend on data that is accurate, well-labeled, and representative of your task. For narrow, well-defined tasks, a smaller model customized on clean, task-specific data can often match or exceed the accuracy of a larger general-purpose model prompted into shape, at a lower cost per inference.</p> <p>In practice, though, preparing that data is often the slowest part of the project. Labeled examples are scarce or expensive, and privacy rules limit what you can use. Quality problems such as duplicates, label noise, and answer leakage go unnoticed until a training run disappoints, and every training technique expects data in its own format. As a result, data scientists spend weeks wrangling data instead of building models.</p> <p>Today, we’re expanding Amazon SageMaker AI with new AI data preparation capabilities for model development workflows. You can generate synthetic examples, assess and improve existing datasets, and bring expert feedback into the process through Python SDK APIs and the AWS AI-ML agent skill. You work in your notebook or IDE, with specifications and code that you can inspect and change.</p> <h2 id="prepare-data-without-hand-building-pipelines">Prepare data without hand-building pipelines</h2> <p>With AI Data Preparation capabilities, you don’t need to stitch together labeling tools, generation scripts, and validation jobs. Each capability is a standalone building block. You can use one of them on its own or chain them together according to the workflow you want.</p> <ul> <li><strong>Auto-labeling:</strong> Label data at scale from an unlabeled dataset, a small labeled seed set, or no labels at all. Built-in task types include classification, named entity recognition, extractive question answering, ordinal rating, and summarization, and you can define custom tasks. You describe the task in a short labeling spec, which is validated against your own examples before the full dataset is labeled. For tasks with a fixed set of possible answers, such as classification, named entity recognition, and extractive question answering, a jury of several models labels each row, and their level of agreement becomes that row’s confidence score.</li> <li><strong>Human review:</strong> Rows below your confidence threshold go to a review queue in Amazon SageMaker Studio, which uses the open-source Label Studio Community Edition. Outside subject matter experts can access the review app through the link in the partner app box in SageMaker Studio. Reviewer corrections feed back into auto-labeling, and you can generate consensus metrics and labeling reports.</li> <li><strong>Synthetic data generation:</strong> Generate training, evaluation, and test data from a specification. You can ground the output in your own seed data and documents, and the service generates a code-based data generation pipeline that you can review, modify, reuse, and scale.</li> <li><strong>Data quality evaluation:</strong> Check datasets against your own business rules, built-in metrics, and LLM-as-a-judge metrics. Find duplicates and near-duplicates, and surface issues you didn’t think to check for, such as label noise or answer leakage.</li> <li><strong>Data transformation:</strong> Clean and reshape data for your target training technique. Describe the change in plain language and have the code generated for you, use built-in operations such as dedupe, merge, split, and filter, or run your own Python at scale.</li> </ul> <p>You work wherever you already work. Use the SageMaker Python SDK directly or provide your coding agent the AI-ML skill in the agent toolkit and let your agent invoke the APIs according to established data science best practices. Describe what you need, and the agent selects and runs the right capability in SageMaker Studio JupyterLab, Kiro, or your local IDE with your coding assistant of choice.</p> <p>For teams building specialized models, the hardest part is often finding enough task-specific data. Splunk, a Cisco company, faced exactly this when building evaluation models for its Agent Observability product.</p> <blockquote> <p><em>“For Splunk Agent Observability, our ability to train robust GenAI Eval models relies on access to high-quality, diverse datasets which are hard to curate for very specific tasks. Amazon SageMaker AI data preparation capabilities for synthetic data generation is meaningfully helping us scale these capabilities. We’ve been able to generate statistically representative synthetic datasets reliably and fast, while preserving the underlying patterns our models depend on. The seamless integration with the rest of our SageMaker workflow means our data scientists and customers spend less time wrangling data and more time building models that drive real enterprise value through Splunk.”</em></p> <p>— Yash Sheth, Director of Engineering, Splunk Agent Observability</p> </blockquote> <h2 id="chain-capabilities-from-raw-data-to-training">Chain capabilities from raw data to training</h2> <p>The value grows when you combine the capabilities. Some common patterns:</p> <p>Build a golden evaluation dataset: Auto-label your raw data from a small seed set. High-confidence rows are accepted automatically, and low-confidence rows go to human reviewers. Their corrections improve the next labeling pass, resulting in a trusted, human-validated dataset to benchmark your models against.</p> <p>Go from raw documents to a trained model in one workflow:</p> <ol type="1"> <li><strong>Generate data:</strong> Synthetic data generation creates task-specific examples grounded in your documents and seed data.</li> <li><strong>Transform it:</strong> Data transformation reshapes the data into the format your training technique expects, s

  • You can generate synthetic examples, assess and improve existing datasets, and bring expert feedback into the process through Python SDK APIs and the AWS AI-ML agent skill.
Nguyên văn nguồn
Chi tiết nguồn
Đánh giá

Ian Buck Kicks Off NVIDIA’s Week at SC26 as Science Enters the Age of Agents

NVIDIA’s Ian Buck kicked off the company’s week at SC26 in Chicago on Monday with new tools for AI agents, software that can plan experiments, run code and check results on researchers’ own systems. In NVIDIA’s special address, Buck announced Nemotron for Science , the Rubin HPC GPU and an expanded Vera CPU platform for high-performance computing . He also detailed a new collaboration in quantum computing with IBM and Fermi National Accelerator Laboratory . “The frontier of computing and the new scientific stack is going to be simulation, as always, but combined with AI and the future of quantum,” said Buck, vice president of hyperscale and high-performance computing at NVIDIA. Buck opened the talk by looking back to his first public talk on CUDA, at an SC06 tutorial in Tampa, Florida, 20 years ago. NVIDIA now offers more than 1,000 CUDA-X libraries , he said. Open Models for Science Agents “[Agents] have been just a game changer in how we interface with our supercomputers, with our devices, our instruments and how we can do the research,” Buck said. Simulation remains the foundation of scientific computing, “which will always be on the top,” Buck said, with AI, quantum and now agents working alongside it. Nemotron for Science combines the Nemotron 3.5 Super VL open model with a new open chemistry dataset and post-training recipes for life sciences and chemistry, Buck explained. Researchers can run it locally on a DGX Spark, a workstation or their own supercomputer. At Argonne National Laboratory, Buck said, an agent built with the toolkit took a single prompt, planned and tested hypotheses, chose which simulations to run and identified two new binding sites. “You own your data. You own the model,” Buck said. Buck highlighted other agents already in use: GROMACS: An agent from KTH Royal Institute of Technology repairs broken GROMACS setups on its own. In one overnight run on a single DGX Spark, it fixed 81% of test cases. SETI Institute: At the Allen Telescope Array, an agent took a one-line prompt to plot a methanol line. Buck explained that it then retuned for Doppler shift on its own [NGC 869 vs. 896, check slide]. Nsight AI Porting Agent: NVIDIA’s new agent ported a routine from the Weather Research and Forecasting (WRF) model to GPUs, Buck said, speeding it up more than 5x using about 150 million tokens. CPUs for the Agent Loop Agents lean on CPUs to run tool calls between model steps, Buck explained. “That’s exactly what agents need,” he said. “They need a CPU that can run, evaluate, run the tool, perform the operations, gather the results.” DKRZ, the German Climate Computing Center, has selected Vera CPUs for its HLRE-5 supercomputer, expected in early 2028. Buck said it joins Vera systems already announced at Texas Advanced Computing Center. A New GPU for Science Rubin HPC is a new GPU in the Vera Rubin platform, built for science, engineering, analytics and AI, Buck said. It supports precisions from FP64 to FP4, adds sparse-data acceleration and connects over seventh-generation NVIDIA NVLink. With more than 50,000 CUDA cores, it has about three times as many as Hopper, Buck said. It is expected in 2028. Quantum Error Correction Buck said a simulation on Jülich’s JUPITER supercomputer showed that a technique called magic state cultivation can cut the operations needed for one logical step tenfold. The work is being added to CUDA-Q. NVIDIA, IBM and Fermilab announced plans today for a quantum computing co-design platform to support error-correction and applications research. NVIDIA and IBM are pairing NVIDIA DGX B200 systems with IBM quantum computers, including those powered by IBM Quantum Nighthawk. Buck also introduced a new breakthrough for quantum decoding. The goal, Buck explained, is to cut decoding latency from about 300 microseconds to 10. A new CUDA-Q decoding algorithm, TCBP-JOBS, cuts error-correction research computation from 12 weeks to 45 minutes, he said. Networking Buck also showed one of the first deployments of NVIDIA Photonics co-packaged optics switches, at Lambda . He said they are 15x more reliable on bit errors and link flaps. See notice regarding software product information.

  • NVIDIA’s Ian Buck kicked off the company’s week at SC26 in Chicago on Monday with new tools for AI agents, software that can plan experiments, run code and check results on researchers’ own systems.
Nguyên văn nguồn
Chi tiết nguồn
Tác tử

Why we rebuilt LangSmith around the agent

We reorganized LangSmith around a single agent record. Build, run, improve, and govern every agent from one place, on any framework or runtime.

  • Build, run, improve, and govern every agent from one place, on any framework or runtime.
Nguyên văn nguồn
Chi tiết nguồn
101–120 / 680
Trang 6 / 34