The Iron(wood) Throne: Google's TPU Stakes Its Claim on Inference
In November 2025, we published a newsletter saying that Google is selling TPUs to external customers. Since then, we have been proven right over and over again with Oracle, OpenAI , Anthropic, and Meta all becoming customers of TPUs just as our Accelerator Model reported. Hardware demand is only half the story, as NVIDIA’s moat has historically been its software and large developer ecosystem. This is where TPU is also making headway, partnering with upstream maintainers across the entire ecosystem, including the Meta PyTorch team, vLLM (Inferact, RedHat), SGLang (RadixArk), llm-d, Ray, etc. This is not a surprise to folks that have read our September 2026 article laying out Google TPU’s externalization strategy . Back in our September article, we published the first third-party TPUv7 results on InferenceX on single-turn workloads, showing that in an apples-to-apples FP8 comparison on Qwen 3.5 397B, TPUv7 Ironwood) delivered up to 50% better performance per dollar than B200 and B300. We also reported that many Google TPU customers were asking for AgentX results, since long-context, multi-turn agentic serving is where inference spend is heading. Previously, Jensen went on the Dwarkesh Podcast and commented on the fact that no other chips were taking advantage of their opportunity to show performance on InferenceX. Source: Dwarkesh (Dylan’s Cousin) Podcast Six months later, and indeed Google is taking full advantage of this opportunity. Today we are releasing the first official long-context, multi-turn, realistic agentic inference benchmark results on TPUv7 across four models, showing that TPUv7 is either competitive on perf per dollar for external customers or delivers the best state-of-the-art result: Muse Spark Open SGLang AgentX (NVFP4 weights): Competitive TPU performance per dollar result even though the architecture was co-designed for GPUs. Submitted by OSS ecosystem developer, RadixArk (Makers of SGLang). GLM-5.2 vLLM AgentX (FP8): TPUv7 is the cheapest FP8 option, delivering 1.2x to 2.7x the tokens per dollar of B200, B300, and MI355X from 60 to 300 tok/s/user. NVIDIA NVFP4 still beats TPUv7 FP8, but with accuracy / quality loss. Qwen3.5 vLLM AgentX (FP8): With llm-d disaggregated serving, TPUv7 beats GB300 FP8 at every interactivity level and restores parity even against GB300 FP4 at higher quality / accuracy. Kimi K3 vLLM AgentX (MXFP4 weights): Competitive with GB300 NVL72 at launch; With TPUv7 delivering 1.15x to 2.5x the tokens per dollar of GB200, B200, B300, and MI355X. Submitted by vLLM OSS ecosystem developer, Inferact. This was benchmarked on InferenceX/AgentX, the industry-standard inference benchmark supported by almost every major buyer of compute, from Microsoft Azure to Oracle to Meta and many more. Furthermore, it has the support of the ML community, including vLLM, LMCache, SGLang, PyTorch, and Huggingface, and the support of major labs like OpenAI, MiniMax, ZAI, Qwen, Moonshot Kimi, etc. Google potential TPU customers and current TPU customers are asking Google for AgentX results because of how realistic the AgentX benchmark is and how similar it is to their real inference workload. For our SemiAnalysis supply chain shipment estimates on TPUv8i, TPUv8t, TPUv9, TPUv10, check our Accelerator Model for full quarter by quarter breakdown of shipments. Source: InferenceX Star the InferenceX GitHub repository if you find the open-source benchmark and data useful!. InferenceX is the only inference benchmark in the world to have TPUv7, OpenAI Jalapeño, Nvidia Rubin NVL72, AMD, and soon, SambaNova and Trainium2/3. Source: InferenceX Recommendation to Jensen NVIDIA only occasionally loses to AMD on some model/scenarios , but TPUs delivering better performance per dollar or competitive performance per dollar is a new experience that NVIDIA senior leadership is unwilling to accept. As soon as TPU posted its first AgentX result on Qwen3.5 397B FP8 showing that even TPUv7 FP8 could beat GB300 NVL72 FP4, NVIDIA tasked a strike force of human engineers and agent swarms to work 997, late into the night until 2am local time and on weekends, across USA and mainland China. Unfortunately for NVIDIA, their first GB300 FP4 counter submission is still 1.64x worse per dollar than TPUv7 FP8 for external customers across most of the curve. We do think that with enough agent swarms, NVIDIA could regain a slight performance lead on GB300 NVL72 FP4 vs TPUv7 FP8, but on NVIDIA FP8 versus TPUv7 FP8, we believe that it is not possible for NVIDIA to regain SOTA for certain models. One thing we hear a lot from open-source maintainers in PyTorch and NVIDIA customers is that there is a lot of bureaucracy; simple things, like adding a new feature to the NVIDIA driver to support a certain feature for PyTorch CudaGraph capture, have been stuck in “internal NVIDIA legal review” for three months. It makes little sense for lawyers to be the blocker for technical driver features. We have seen similar challenges when trying to get NVIDIA to add features like improving ConnectX, NVSwitch, HGX Board (Vulcan, Umbriel) DCGM power telemetry, where it takes a while just to get the timeline for the timeline. In comparison, startup mindset companies like AMD (under the leadership of Anush, Ramine, Vamsi, Lisa) can rapidly develop these features. There needs to be an cultural shift in massively reducing bureaucracy at NVIDIA. This is one of the biggest risks to NVIDIA Moat that Jensen has to tackle. In our experience and many customers experience, Nvidia is now slower moving than AMD, Google, and Amazon to external suggestion, git commits, and feedback. That being said, certain teams at NVIDIA are cutting through the bureaucracy, such as the NVIDIA ex-CentML vLLM team . AMD and TPUs added innovative Megakernel decode support to open source popular production inference engine . Within a week, they were able to defend the NVIDIA by rapidly adding and merging Megakernel support too (PR approved by AMD vLLM maintainer).
- We also reported that many Google TPU customers were asking for AgentX results, since long-context, multi-turn agentic serving is where inference spend is heading.
- Today we are releasing the first official long-context, multi-turn, realistic agentic inference benchmark results on TPUv7 across four models, showing that TPUv7 is either competitive on perf per dollar for external customers or delivers the best state-of-the-art result: Muse Spark Open SGLang AgentX (NVFP4 weights): Competitive TPU performance per dollar result even though the architecture was co-designed for GPUs.
- Google potential TPU customers and current TPU customers are asking Google for AgentX results because of how realistic the AgentX benchmark is and how similar it is to their real inference workload.
Chi tiết nguồn
- The Iron(wood) Throne: Google's TPU Stakes Its Claim on Inference SemiAnalysis · 6 thg 11, 2026