How Airbnb uses cost per task to operate coding agents on Amazon Bedrock
<p>Amazon Bedrock provides access to foundation models from multiple providers through a common API, with AWS managing the model infrastructure. It integrates with AWS access controls and monitoring, and processes prompts, code, and model responses within AWS.</p> <p>As coding agents move into production, changes in models and usage can affect latency, capacity, and cost. Amazon Bedrock runtime metrics in Amazon CloudWatch help teams investigate latency increases, throttling, and changes in token consumption. Evaluations, or evals, complement these operational metrics by assessing the quality of an agent’s work on a fixed set of representative tasks.</p> <p>In this post, we show how Airbnb evaluates models for coding agents and monitors their use on Amazon Bedrock. We explain how cost per task guides model selection and how production data helps us revisit those choices as usage changes. We also describe the runtime signals used to investigate latency, throttling, and spend.</p> <p>At Airbnb, thousands of engineers and engineering-adjacent people use AirChat, our internal tool for running coding agents, every week. The agents send their model requests through our large language model (LLM) gateway, which runs on Amazon Bedrock. Bedrock processes Airbnb’s code, prompts, and model responses within AWS. Eight new models came out in about five weeks. By mid-October, about two thirds of the requests our engineers sent through coding agents went to those new models.</p> <p>Each release can change output tokens, cost, latency and capacity at once, and the benchmark charts published with a launch don’t show how a model handles our engineers’ work. So we run two systems side by side. Before a rollout, we run evals on our own tasks to measure each model’s score and cost per task. Cost per task is the list-price cost of all the tokens an agent uses to finish one task.</p> <p>After a rollout, observability shows us what changed in production. When production usage doesn’t match the eval results, we find out why.</p> <h2 id="solution-overview">Solution overview</h2> <p>You can diagnose most production problems with a few signals in the <code>AWS/Bedrock</code> namespace in CloudWatch:</p> <ul> <li>Latency: <code>InvocationLatency</code>, <code>TimeToFirstToken</code>, <code>OutputTokenCount</code>, and output tokens per second (OTPS), which you derive.</li> <li>Throttling: <code>InvocationThrottles</code>, plus the tokens each request reserves against your quota, which you publish yourself. Bedrock also publishes <code>EstimatedTPMQuotaUsage</code>, which measures your estimated tokens per minute (TPM) quota consumption, including cache write tokens and output burndown multipliers, across all inference APIs (<code>Converse</code>, <code>InvokeModel</code>, <code>ConverseStream</code>, and <code>InvokeModelWithResponseStream</code>). You can use this metric to set proactive alarms before reaching your quota limit, track your quota consumption across your models, and request further quota increases before usage is rate limited.</li> <li>Cost: <code>InputTokenCount</code> (uncached input), <code>CacheReadInputTokenCount</code>, <code>CacheWriteInputTokenCount</code> and <code>OutputTokenCount</code>, each priced at its own rate. Cache reads cost less than uncached input, and cache writes can cost more depending on the model. A lower cache hit rate can increase cost per request.</li> </ul> <p>On top of these metrics, we add cost per task. Our evals measure a score and a cost per task for every model and every reasoning effort we allow. Reasoning effort is a setting that controls how much a model reasons before it responds. We use the results to choose each default, which is the model and effort that our coding tools use unless an engineer picks another, and each effort ceiling, which is the highest effort we allow for a model.</p> <h2 id="architecture">Architecture</h2> <p>Engineers run AirChat on their AirDev workspaces, which are personal development machines in the cloud. They can also hand a task to AirChat Remote, which runs cloud agents in separate cloud workspaces while the engineer does something else. Both paths go through our LLM gateway to Amazon Bedrock.</p> <p>Figure 1 shows how AirChat and AirChat Remote reach Amazon Bedrock through our gateway. It also shows how evals and production monitoring guide model defaults and effort ceilings.</p> <div style="width: 810px" class="wp-caption alignnone"> <a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/18/ML-22452-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/18/ML-22452-1.png" alt="AirChat and AirChat Remote reach Amazon Bedrock through Airbnb’s gateway, with CloudWatch metrics and eval results setting model defaults and effort ceilings" width="800"></a> <p class="wp-caption-text">Figure 1: Coding agents and cloud agents call Amazon Bedrock through our LLM gateway. Our usage records and CloudWatch metrics show real use, and we use eval results to set defaults and effort ceilings</p> </div> <p>The gateway records the model, client, team, and user for each request. We use these records only in aggregate, for cost and capacity planning. CloudWatch supplies token counts, latency, errors, and throttles. We compare those production signals with our eval scores and cost per task to decide when defaults or effort ceilings need to change.</p> <h2 id="prerequisites">Prerequisites</h2> <p>On the AWS side, you need:</p> <ul> <li>Access to the models in Amazon Bedrock you plan to run and evaluate. For model availability by AWS Region, see <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/models-region-compatibility.html" target="_blank" rel="noopener">Regional availability by models</a> in the Amazon Bedrock User Guide.</li> <li>Streaming calls through <code>ConverseStream</code> or <code>InvokeModelWithResponseStrea
- </p> <p>In this post, we show how Airbnb evaluates models for coding agents and monitors their use on Amazon Bedrock.
- png" alt="AirChat and AirChat Remote reach Amazon Bedrock through Airbnb’s gateway, with CloudWatch metrics and eval results setting model defaults and effort ceilings" width="800"></a> <p class="wp-caption-text">Figure 1: Coding agents and cloud agents call Amazon Bedrock through our LLM gateway.
- Our usage records and CloudWatch metrics show real use, and we use eval results to set defaults and effort ceilings</p> </div> <p>The gateway records the model, client, team, and user for each request.
Chi tiết nguồn
- How Airbnb uses cost per task to operate coding agents on Amazon Bedrock AWS Machine Learning Blog · 19 thg 11, 2026