AllyshipContent Radar

Deep dives 738

Paper và bài viết kỹ thuật

Crawl live-1795068900000 · 13:22 19 thg 11, 2026 · Xuất bản 13:30 19 thg 11, 2026

Đánh giá

The Iron(wood) Throne: Google's TPU Stakes Its Claim on Inference

In November 2025, we published a newsletter saying that Google is selling TPUs to external customers. Since then, we have been proven right over and over again with Oracle, OpenAI , Anthropic, and Meta all becoming customers of TPUs just as our Accelerator Model reported. Hardware demand is only half the story, as NVIDIA’s moat has historically been its software and large developer ecosystem. This is where TPU is also making headway, partnering with upstream maintainers across the entire ecosystem, including the Meta PyTorch team, vLLM (Inferact, RedHat), SGLang (RadixArk), llm-d, Ray, etc. This is not a surprise to folks that have read our September 2026 article laying out Google TPU’s externalization strategy . Back in our September article, we published the first third-party TPUv7 results on InferenceX on single-turn workloads, showing that in an apples-to-apples FP8 comparison on Qwen 3.5 397B, TPUv7 Ironwood) delivered up to 50% better performance per dollar than B200 and B300. We also reported that many Google TPU customers were asking for AgentX results, since long-context, multi-turn agentic serving is where inference spend is heading. Previously, Jensen went on the Dwarkesh Podcast and commented on the fact that no other chips were taking advantage of their opportunity to show performance on InferenceX. Source: Dwarkesh (Dylan’s Cousin) Podcast Six months later, and indeed Google is taking full advantage of this opportunity. Today we are releasing the first official long-context, multi-turn, realistic agentic inference benchmark results on TPUv7 across four models, showing that TPUv7 is either competitive on perf per dollar for external customers or delivers the best state-of-the-art result: Muse Spark Open SGLang AgentX (NVFP4 weights): Competitive TPU performance per dollar result even though the architecture was co-designed for GPUs. Submitted by OSS ecosystem developer, RadixArk (Makers of SGLang). GLM-5.2 vLLM AgentX (FP8): TPUv7 is the cheapest FP8 option, delivering 1.2x to 2.7x the tokens per dollar of B200, B300, and MI355X from 60 to 300 tok/s/user. NVIDIA NVFP4 still beats TPUv7 FP8, but with accuracy / quality loss. Qwen3.5 vLLM AgentX (FP8): With llm-d disaggregated serving, TPUv7 beats GB300 FP8 at every interactivity level and restores parity even against GB300 FP4 at higher quality / accuracy. Kimi K3 vLLM AgentX (MXFP4 weights): Competitive with GB300 NVL72 at launch; With TPUv7 delivering 1.15x to 2.5x the tokens per dollar of GB200, B200, B300, and MI355X. Submitted by vLLM OSS ecosystem developer, Inferact. This was benchmarked on InferenceX/AgentX, the industry-standard inference benchmark supported by almost every major buyer of compute, from Microsoft Azure to Oracle to Meta and many more. Furthermore, it has the support of the ML community, including vLLM, LMCache, SGLang, PyTorch, and Huggingface, and the support of major labs like OpenAI, MiniMax, ZAI, Qwen, Moonshot Kimi, etc. Google potential TPU customers and current TPU customers are asking Google for AgentX results because of how realistic the AgentX benchmark is and how similar it is to their real inference workload. For our SemiAnalysis supply chain shipment estimates on TPUv8i, TPUv8t, TPUv9, TPUv10, check our Accelerator Model for full quarter by quarter breakdown of shipments. Source: InferenceX Star the InferenceX GitHub repository if you find the open-source benchmark and data useful!. InferenceX is the only inference benchmark in the world to have TPUv7, OpenAI Jalapeño, Nvidia Rubin NVL72, AMD, and soon, SambaNova and Trainium2/3. Source: InferenceX Recommendation to Jensen NVIDIA only occasionally loses to AMD on some model/scenarios , but TPUs delivering better performance per dollar or competitive performance per dollar is a new experience that NVIDIA senior leadership is unwilling to accept. As soon as TPU posted its first AgentX result on Qwen3.5 397B FP8 showing that even TPUv7 FP8 could beat GB300 NVL72 FP4, NVIDIA tasked a strike force of human engineers and agent swarms to work 997, late into the night until 2am local time and on weekends, across USA and mainland China. Unfortunately for NVIDIA, their first GB300 FP4 counter submission is still 1.64x worse per dollar than TPUv7 FP8 for external customers across most of the curve. We do think that with enough agent swarms, NVIDIA could regain a slight performance lead on GB300 NVL72 FP4 vs TPUv7 FP8, but on NVIDIA FP8 versus TPUv7 FP8, we believe that it is not possible for NVIDIA to regain SOTA for certain models. One thing we hear a lot from open-source maintainers in PyTorch and NVIDIA customers is that there is a lot of bureaucracy; simple things, like adding a new feature to the NVIDIA driver to support a certain feature for PyTorch CudaGraph capture, have been stuck in “internal NVIDIA legal review” for three months. It makes little sense for lawyers to be the blocker for technical driver features. We have seen similar challenges when trying to get NVIDIA to add features like improving ConnectX, NVSwitch, HGX Board (Vulcan, Umbriel) DCGM power telemetry, where it takes a while just to get the timeline for the timeline. In comparison, startup mindset companies like AMD (under the leadership of Anush, Ramine, Vamsi, Lisa) can rapidly develop these features. There needs to be an cultural shift in massively reducing bureaucracy at NVIDIA. This is one of the biggest risks to NVIDIA Moat that Jensen has to tackle. In our experience and many customers experience, Nvidia is now slower moving than AMD, Google, and Amazon to external suggestion, git commits, and feedback. That being said, certain teams at NVIDIA are cutting through the bureaucracy, such as the NVIDIA ex-CentML vLLM team . AMD and TPUs added innovative Megakernel decode support to open source popular production inference engine . Within a week, they were able to defend the NVIDIA by rapidly adding and merging Megakernel support too (PR approved by AMD vLLM maintainer).

  • We also reported that many Google TPU customers were asking for AgentX results, since long-context, multi-turn agentic serving is where inference spend is heading.
  • Today we are releasing the first official long-context, multi-turn, realistic agentic inference benchmark results on TPUv7 across four models, showing that TPUv7 is either competitive on perf per dollar for external customers or delivers the best state-of-the-art result: Muse Spark Open SGLang AgentX (NVFP4 weights): Competitive TPU performance per dollar result even though the architecture was co-designed for GPUs.
  • Google potential TPU customers and current TPU customers are asking Google for AgentX results because of how realistic the AgentX benchmark is and how similar it is to their real inference workload.
Nguyên văn nguồn
Chi tiết nguồn
Tác tử

Formally verifying agent-written full-stack web apps with PDD Kit

Câu hỏi So, how do you reliably scale review as agents generate increasingly large amounts of code?

At a glance Formal methods for everyday software. PDD Kit helps you turn your intent into a formal spec, then proves the code your agent writes for every input and uses tests where a proof can’t reach. It takes formal methods beyond niche, specialist work to engineers without a formal methods background and to everyday applications, like the full-stack web app shown here. Exhaustive line-by-line human review can become difficult to scale as AI agents generate increasingly large amounts of code. The plan is supposed to be the contract between you and a coding agent, but nothing actually forces the agent to follow it. The tests it writes usually check what it built, rather than what you asked for. PDD Kit makes the agent keep the plan. It works with you to turn your intent into a formal specification in Verus (opens in new tab) , proves the agent’s code meets that spec for every input, and falls back to tests where a proof can’t reach. We tried it on a full-stack web app. Inside GitHub Next’s Chopin (opens in new tab) , we used PDD Kit to formally verify the login logic of Pet Store, a Rust (opens in new tab) storefront. Verus hardened the code and uncovered existing bugs, proving the absence of login bugs with respect to the specification. The developer reviewed the intent and never read the generated code line-by-line. The plan is the contract the agent should follow, but nothing constrains the agent to follow it. Coding agents now ship whole features in one sitting often spanning tens of thousands or hundreds of thousands of lines of code. As a result, reading every line of the pull request doesn’t scale to this new way of work. Now, most teams invest in the plan and relevant testing instead: describe to an agent what they want, go back and forth until the plan and proposed tests look right, and hand them off to the agent for implementation. This plan or natural language spec is meant to be the contract that an agent holds. However, in practice, the agent can drift from it. It implements something slightly different, then writes tests that pass for its implementation rather than the code you intended it to write. Continuous integration (CI) grows due to a profusion of low quality and high quantity of tests, coverage looks healthy, and the gap between what you asked for and what shipped stays hidden until a bug report or a nasty edge case shows up. So, how do you reliably scale review as agents generate increasingly large amounts of code? Agent code fails in two ways a careful reviewer would usually catch: it leaves out something obviously necessary, or it does something obviously wrong. Tests alone don’t close that gap, because they only exercise a tiny slice of what a program can do. Closing it needs a different kind of machinery, which is where proofs come in. Review asks “does this look right?” A proof answers “is this right?” A mathematical proof shows that code meets its specification for every input, including the ones your test suite can miss. That provides stronger assurance for specified properties than testing alone, especially when line-by-line review isn’t practical. Proofs used to be too expensive for everyday software. You needed specialists, unfamiliar languages, proofs many times larger than the code, and proofs took years to develop, but two things have changed since. First, recent advances in coding agents have made it increasingly practical for them to help generate formal proofs from well-defined specs. Secondly, agents can now better help you write formal specs. We call this new software development paradigm Proof-Driven Development (PDD). PDD Kit is our toolkit for it. How PDD Kit works PDD Kit is to formal methods roughly what Spec Kit (opens in new tab) is to Spec-Driven Development. It runs as a partner inside Chopin (opens in new tab) , GitHub Next’s real-time, multiplayer planning environment, where a team and agents can plan in one shared document. The workflow has two halves (Figure 1). First, team members review intent in plain language. PDD Kit asks the questions a careful reviewer would ask, explains the relevant code as it goes, and records the answers in an intent artifact , a document that bridges the user’s intent with the formal specification or tests. The parts of the user’s intent that can be formalized are converted into formal specifications and the rest, which must be taken on trust, are converted to informal specifications for tests. PDD Kit automatically generates proofs that the code satisfies the derived formal specs and high-quality tests for the informal specs. You don’t need a formal methods background for any of it. Figure 1. Proof-Driven Development with PDD Kit. Team members settle intent. Verus and the test suite hold the code to it. Consequently, every result in PDD Kit says which kind of verification it rests on: Formal verification is a mathematical proof that code satisfies its specification for every possible input, checked by a tool called a verifier. For PDD Kit today, that verifier is Verus (opens in new tab) , a proof language for Rust co-developed by Microsoft Research, Carnegie Mellon University, VMware Research, ETH Zurich, MPI-SWS, UNSW Sydney, the University of Texas at Austin, the University of Michigan, the University of British Columbia, the University of Wisconsin–Madison and the Weizmann Institute of Science. Empirical verification checks code by running it and observing what it does. It covers what a proof can’t reach, such as a third-party cryptography library or a user interface. It uses unit and end-to-end tests, terminal automation, browser agents, and computer-use agents for anything a browser can’t drive. A passing test only shows the code worked on the inputs it tried, so PDD Kit labels it as evidence rather than proof. The line between the two is the Trusted Computing Base (TCB) : the code that the proof assumes behaves as described. PDD Kit makes that line explicit and asks you to sign off on each piece. T

  • As a result, reading every line of the pull request doesn’t scale to this new way of work.
  • Consequently, every result in PDD Kit says which kind of verification it rests on: Formal verification is a mathematical proof that code satisfies its specification for every possible input, checked by a tool called a verifier.
Nguyên văn nguồn
Chi tiết nguồn
Đánh giá

A Model For Everyone: How Ronak From Trajectory Is Building Products For Continual Learning

Câu hỏi Hosted by Pat Grady and Lauren Reeder, Sequoia Capital 0:00 – Introduction 0:45 – Pseudo continual learning: paper maps vs Google Maps 2:48 – Do we need new architectures?

Today’s agents are stuck in what Ronak Malde calls “pseudo continual learning”: intelligent, but updated in offline batches, like a paper road map you can’t reroute mid-trip. Ronak, co-founder of Trajectory, explains why closing that loop is the next learning paradigm, and why training your own model, which used to take a dedicated research team, months of work and hundreds of millions in compute, is now something a company can do in a few clicks with their own data. He details two research bets they are making available to everyone with their product: 1) a variant of GRPO that removes the need for a simulated environment by learning from single real-world rollouts instead of 16 replays of the same task, and 1) a meta-grader that increased data efficiency by 100x. Ronak explains how Harvey trained a model six times cheaper than frontier, improving an open-source baseline by over 4x on Harvey's own legal benchmark. He defines the difference between continual learning and recursive self-improvement and predicts product builders will become gardeners, setting trellises for products that grow themselves. Hosted by Pat Grady and Lauren Reeder, Sequoia Capital 0:00 – Introduction 0:45 – Pseudo continual learning: paper maps vs Google Maps 2:48 – Do we need new architectures? Rich Sutton, David Silver, and the missing layer 4:33 – From a $100M consulting model to a product: lessons from SWE-1 at Windsurf 7:36 – Harvey: 6x cheaper than frontier, 4% to 18% on its own benchmark 8:55 – Why "Trajectory": the unit of work is a task, not a token 10:13 – The primitives of continual learning: traces, replayable environments, reward 12:56 – Group Size 1 and the meta-grader: training without an environment or a labeled reward 16:16 – The platform: train, eval and deploy in one click, and who it's for 18:58 – Rogo and intelligence density 20:14 – What's next: curiosity, and why continual learning beats RSI on alignment 24:19 – Research and product under one roof, and the product builder as gardener 27:26 – Why work at Trajectory, and the optimistic case: from 1% to 99%

  • Ronak explains how Harvey trained a model six times cheaper than frontier, improving an open-source baseline by over 4x on Harvey's own legal benchmark.
  • He defines the difference between continual learning and recursive self-improvement and predicts product builders will become gardeners, setting trellises for products that grow themselves.
Nguyên văn nguồn
Chi tiết nguồn
Đánh giá

Clearing up a robot’s fuzzy memory

<p>In an emergency, a human might someday ask a robotic assistant to fetch a wall-mounted defibrillator located down the hall. But if the robot has incorrectly labeled the defibrillator as a fire extinguisher in its stored memory, it may not know where to find the life-saving device.</p><p>This robotic assistant could benefit from a new memory system created by MIT researchers. The system identifies entries in a robot’s memory that might be wrong and corrects the descriptions of mislabeled objects.&nbsp;</p><p>In the future, this method could make household or industrial robots better able to perform complex, multistep tasks, such as clearing a dining room table or working alongside human collaborators to assemble a piece of machinery.&nbsp;</p><p>Called UQ-DAAAM (which stands for Uncertainty Quantification for Describe Anything, Anywhere, at Any Moment), the system compares multiple descriptions of the same object, seen from different viewpoints, to quantify the robot’s uncertainty about what it saw.&nbsp;</p><p>It then selects additional views to reduce the uncertainty and combines these captions into a more accurate description of the object, which it saves to memory.</p><p>The researchers also employed a special technique to streamline this process so a mobile robot can continually update uncertain memory entries in real time.&nbsp;</p><p>When compared to baseline methods, UQ-DAAAM boosted the robot’s confidence and improved the accuracy of its answers to questions about objects, their locations, and when it saw them.&nbsp;</p><p>“Contrary to ChatGPT, when we think about physical AI, it is not enough to simply put a disclaimer cautioning that sometimes the robot might be wrong. Allowing a robot to know what it doesn’t know is going to be essential for this technology to be deployed safely and have a positive impact on society,” says Luca Carlone, an associate professor in MIT’s Department of Aeronautics and Astronautics (AeroAstro), principal investigator in the Laboratory for Information and Decision Systems (LIDS), director of the MIT SPARK Laboratory, and the senior author of the <a href="https://arxiv.org/pdf/2606.08277" target="_blank">paper</a> presenting UQ-DAAAM.</p><p>He is joined on the paper by co-lead authors and MIT graduate students Harry Zhang and Nicolas Gorlo. The research will be presented at the Conference on Neural Information Processing Systems.&nbsp;</p><p><strong>Amending memory</strong></p><p>This new method builds on the researchers’ prior work,&nbsp;<a href="https://news.mit.edu/2026/could-ai-tell-you-where-you-left-your-keys-0617" target="_blank">DAAAM</a>, a robot memory system that uses a vision-language model (VLM) to richly describe objects in scenes a robot captures with its onboard cameras. These descriptions are stored as detailed captions in a 3D map-based representation of the robot’s environment that it can call upon later to answer questions.&nbsp;</p><p>But the robot sometimes sees objects only partially or from far away, or its camera might capture blurry or poorly lit images. In these cases,&nbsp;DAAAM might describe these objects incorrectly, such as by mistaking a gray chimney for a squirrel perched on a rooftop.</p><p>“With no way to correct itself, this wrong memory would persist as the robot proceeds in its environment, which could impact downstream applications,” Zhang says.</p><p>UQ-DAAAM integrates uncertainty quantification into the DAAAM framework. It allows the robot to measure its confidence in the accuracy of object descriptions and fix incorrect captions as it travels through an environment.</p><p>While on the move, the robot constantly collects and stores images of the world around it. Many of these images show the same objects captured from different angles. UQ-DAAAM uses these additional viewpoints to identify and correct unreliable descriptions.</p><p>First, it calculates the VLM’s confidence level in the words it generated to describe a potentially mislabeled object. Then, it measures consistency by comparing this description to captions of images that describe the same object from different vantage points.</p><p>“We fuse both sources of uncertainty into a single number that captures how certain the VLM is in its overall understanding of the object,” Carlone explains.</p><p>If that number exceeds a certain threshold, the uncertainty is too high, and UQ-DAAAM will refine the caption by using additional images of the object.</p><p>It does this using a heuristic, or simple rule, that determines which additional viewpoints will reduce the robot’s uncertainty about the object. For instance, it might choose an image captured a few frames later that clearly shows smoke emanating from the chimney.&nbsp;</p><p>The researchers mathematically proved that this heuristic can identify the best frames to select without rerunning the VLM, dramatically reducing the amount of computation.&nbsp;</p><p>“The key challenge Nico and Harry had to overcome is how to combine VLMs with uncertainty quantification into something that is running in a fraction of a second on a robot that is moving,” Carlone says.</p><p><strong>Remembering with confidence</strong></p><p>Once it has identified additional captions that will reduce its uncertainty about the object, the system uses an LLM to fuse them into a single object description. To boost accuracy, the LLM puts more weight on new captions and those with less ambiguity.</p><p>This refinement step is able to resolve ambiguities for about 92 percent of the objects in the robot’s environment, while only requiring two additional captions per object. It increased the accuracy of object descriptions by 5 percent compared to the state of the art.</p><p>UQ-DAAAM also provided more accurate answers to questions about the objects it observed, where it saw them, and when it last encountered them than baseline methods.</p><p>“Performing this uncertainty quantification procedure across the entire map the robot generate

  • </p><p>It does this using a heuristic, or simple rule, that determines which additional viewpoints will reduce the robot’s uncertainty about the object.
  • </p><p><strong>Remembering with confidence</strong></p><p>Once it has identified additional captions that will reduce its uncertainty about the object, the system uses an LLM to fuse them into a single object description.
Nguyên văn nguồn
Chi tiết nguồn
Kỹ thuật

The Inside Story of the 50 Who Built Modern AI With Kevin Roose

Câu hỏi How did the race to build AGI really unfold, and who made the decisions that shaped it?

How did the race to build AGI really unfold, and who made the decisions that shaped it? Tech journalist Kevin Roose interviewed 150 people at the center of the AI industry to find out. The result is _The AGI Chronicles,_ a riveting account of how modern AI was built and the choices that will determine what comes next. On this week’s episode of _The Every Podcast,_ Natalia Quintero sits down with Roose to talk about how he got a secretive industry to open up, how he did (and didn’t) use AI in his book writing process, and why he sees his job as explaining the “very confusing and insular” world of Silicon Valley to everyone else. Links to resources mentioned in the episode: • Follow Kevin Roose: https://x.com/kevinroose • Get Kevin Roose’s book, _The AGI Chronicles:_ https://theagichronicles.com/ • Machine Gods: https://www.npr.org/podcasts/g-s1-143588/machine-gods • “The Big Blob of Compute Hypothesis” excerpt on Kevin’s Substack: https://kevinroose.substack.com/p/the-essay-that-started-the-ai-race • _Profiles of the Future_ by Arthur C. Clarke: https://www.amazon.com/Profiles-Future-Inquiry-Limits-Possible/dp/0575402776 • _The Making of the Atomic Bomb_ by Richard Rhodes: https://www.amazon.com/Making-Atomic-Bomb-Richard-Rhodes/dp/1451677618 • NotebookLM: https://notebook.google/ Timestamps: 0:00 Introduction 2:23 Are we doomed? Roose on AI risk and the case for mandatory safety testing 7:13 Why Roose raced to write the history of AGI before the records disappeared 10:45 How Roose got 150 sources in a secretive industry to open up 13:49 Dario Amodei’s “Big Blob of Compute” memo 19:12 How Roose’s mother’s book on the survivors of the atomic bomb influenced his work 21:40 The moment Roose first “felt the AGI” 23:24 How a “council of Claudes” fact-checked Roose’s book 31:59 Launching _Machine Gods_ and reinventing video journalism 41:41 Weightlifting versus forklifting: How Roose decides what to hand off to AI

  • The result is _The AGI Chronicles,_ a riveting account of how modern AI was built and the choices that will determine what comes next.
Nguyên văn nguồn
Chi tiết nguồn
Tác tử

How big is the open model economy today?

Câu hỏi The primary question I wanted to address today is: What is the market for open model inference?

My longest running agent testbed has been building the Interconnects Dashboard — an aggregator of all of the best data on open model adoption and performance from across the web. This dashboard is freely available and becoming quite useful. Still, all the measurements we have on open model adoption are incredibly imprecise. We have noisy platform metrics, marketing-related metrics from startups, and imperfect public evaluations. It takes an informed eye to figure out where things are going. These research notes are my current best approximations of where we stand. As with most things on Interconnects, our goal is to build well more value than we capture, so the data on the dashboard will remain free for the foreseeable future. These notes will be more raw than many of my articles, a place for sharing what I’m mulling over but don’t have complete evidence for (yet). They will be primarily paid, enabling me to speak more freely. If you work in AI policy or policy-facing research, please reach out to us and we can likely get discounted or free access. The primary question I wanted to address today is: What is the market for open model inference? Is it growing? Is it a rounding error relative to OpenAI/Anthropic? Share I’ve been building the best measurements I can here, primarily using the open model inference startups Baseten, Fireworks, and Together AI as the bellwethers here (along with the open model share of OpenRouter’s token volume, which they thankfully report in their API). The hyperscalers also likely have a major footprint here, e.g. Amazon Bedrock and Azure’s model garden, but I haven’t seen them break down usage by model category. The startups’ fates are much more directly correlated with the success of open model inference as a category. For the latest data, we can start with a slide from my talk on the open model economy (recording here ). These are the public disclosures from the open model startups on their token volume, over time, log-scale. This shows near-exponential growth, before the uptick that would follow GLM 5.2’s step change in open agents . For example, Fireworks announced at their first company conference this week that they’re serving more than 40T tokens per day to customers’ customized models. This is just one slice of their inference, a vertical that they said grew over 2x from this summer, so I would expect their total number this fall to be well above 100T tokens when we get the next update. My first instinct with these numbers has been to ground them versus any closed model or hyperscale company we can, at least in terms of token volume. Thankfully, Google releases token volumes for the entire platform and API usage in their earnings reports. We can overlay those two numbers on top of the open model inference platforms, with the resulting plot below: Read more

  • We can overlay those two numbers on top of the open model inference platforms, with the resulting plot below: Read more
Nguyên văn nguồn
Chi tiết nguồn
Đánh giá

Use NVIDIA NeMo to Turn Agent Failures into Tests and Harness Fixes

Turn production evidence into prioritized issues and custom evaluations that help you improve prompts, tools, models, memory, skills, and harness code. When you... Turn production evidence into prioritized issues and custom evaluations that help you improve prompts, tools, models, memory, skills, and harness code. When you build an agent for your domain, generic benchmark scores may not explain its failures. An agent might choose the wrong tool, follow the wrong workflow branch, miss relevant context, use an unnecessarily expensive model… Source

  • Turn production evidence into prioritized issues and custom evaluations that help you improve prompts, tools, models, memory, skills, and harness code.
  • Turn production evidence into prioritized issues and custom evaluations that help you improve prompts, tools, models, memory, skills, and harness code.
Nguyên văn nguồn
Chi tiết nguồn
Kỹ thuật

Show HN: Narro – Read web pages and Kindle books using local AI voices

I wanted a read-aloud extension that could read from the Kindle Online Reader. The ones I tried used voices I did not enjoy listening to or had a UX that was too cluttered or annoying or needed a signup. So I decided to vibe code my own. It is open source and uses open weight models that run in the browser. https://swrdfish.github.io/Narro/

HN: open weights · ghoshbinayak 10 lượt bình chọn 1 bình luận Mở bản gốc ↗
Nguyên văn nguồn
Chi tiết nguồn
Đánh giá

⚡️ The 2026 Software Factory Infra Stack — Zach Lloyd, Warp.dev

Câu hỏi How do software factories change the work of product engineers?

Software factories at Warp: founder and CEO Zach Lloyd walks swyx through the system his team uses to turn tasks into pull requests, measure agent quality, and improve the workflow itself. The demo follows a request from Slack to a PR, then explores shared agent traces, cost and throughput scoring, and benchmarks built from real engineering work. Zach reports that tuning models and subagent use brought Warp’s internal cost from about $80 per PR to roughly $15–20, alongside higher throughput. How do software factories change the work of product engineers? What belongs in a benchmark? And how can agents help improve the factory that runs them? Join Zach and the AI engineering community at AIE CODE, November 10–12, 2026 in San Francisco. Apply to attend: https://ai.engineer/code/2026 Chapters 0:00 Introduction: software factories at Warp 0:53 AIE CODE in San Francisco 1:31 Zach Lloyd and software factories 4:30 Define the factory in code 7:34 Follow a Slack request to a PR 8:41 Measure cost, throughput and quality 14:32 Agents improving the factory 16:10 Benchmark your own tasks 23:59 From $80 to $15–20 per PR 27:03 Who needs software factories? 33:23 Product engineers and factory engineers Links • Warp: https://www.warp.dev/ • AIE CODE: https://ai.engineer/code/2026 • Zach Lloyd — Software Engineering Is Becoming Factory Engineering: https://youtu.be/tUPPVhBBcoM • Suraj Gupta — Building Self-Improving Agent Software Factories: https://youtu.be/TN3mj92oZ8I • Latent Space: https://latent.space/ Sponsorship and business inquiries: business@latent.space #SoftwareFactories #Warp #AIEngineering

  • Software factories at Warp: founder and CEO Zach Lloyd walks swyx through the system his team uses to turn tasks into pull requests, measure agent quality, and improve the workflow itself.
Nguyên văn nguồn
Chi tiết nguồn
Huấn luyện

Building connections across the research infrastructure

<p>Running a modern research laboratory requires more than maintaining sophisticated instruments. Researchers need to complete training, obtain access, schedule and operate equipment, navigate billing, and more. For large, shared facilities, coordinating these activities across hundreds of instruments and thousands of users can be a significant undertaking. As research becomes increasingly interdisciplinary, another challenge is emerging: finding the right capability in the first place.</p><p>Addressing these challenges was a central theme of the 2026 NEMO User Meeting, hosted by MIT.nano Oct. 5–6. The meeting brought together 65 representatives from 25 laboratories and organizations, including facility directors, laboratory managers, technical staff, software developers, industry representatives, and government researchers. Sponsored by the Northeast Microelectronics Coalition (<a href="https://nemicroelectronics.org/">NEMC</a>) hub, the meeting focused on sharing approaches to lab management and building connections among facilities.</p><p>At the center of the meeting was&nbsp;the NanoFab Equipment Management and Operations (<a href="https://nanousers.mit.edu/sites/default/files/documents/NEMO%20Quick%20Reference_Users_v1.3.pdf">NEMO</a>) system, an open-source platform that provides tools for managing shared research facilities while allowing individual laboratories to adapt the system to their own needs.&nbsp;The meeting highlighted an important characteristic of the NEMO community: The same platform can serve a very diverse set of facilities, and while the scale or the scientific missions of the facilities may differ, their core operational challenges are often remarkably similar.&nbsp;</p><p>For Anna Osherov,&nbsp;associate director of Characterization.nano and organizer of the meeting, the collaborative model is one of the most compelling aspects of NEMO.</p><p>“Open source changes the way we think about technology development,” she says. “We can build on what others have already developed, contribute improvements back to the community, and collectively create a platform that is more capable than what any one facility could develop on its own.”</p><p>NEMO was originally&nbsp;released as open-source software by&nbsp;the National Institute of Standards and Technology (<a href="https://www.nist.gov/about-nist">NIST</a>) in 2018. Its community edition (NEMO-CE), launched in 2023, allows facilities to develop and share extensions to the platform. At the October meeting, participants from organizations including MIT; Stanford, Columbia, and Princeton universities; the University of Pennsylvania; NIST; NASA’s Jet Propulsion Laboratory; and Atlantis Labs presented tools to address challenges ranging from chemical management and user administration to equipment status, training, and workflow automation.</p><p>The range of presentations illustrated how differently configured facilities are adapting a common platform to their own needs. Stanford, for example, presented real-time displays for equipment status and reservations, while Columbia shared its experience migrating to NEMO and extending the system for billing and facility workflows. Princeton demonstrated automated process charts for monitoring tool performance, and the University of Pennsylvania presented approaches to chemical management.</p><p>Overall, the projects demonstrated the premise behind the NEMO community: a solution developed to address a problem at one facility can become a resource for others.</p><p>“Together we can do way more than any one of us individually can,” said Vladimir Bulović, director of MIT.nano, during his opening remarks. “NEMO is the platform that gave us, finally, a streamlined way to do so.”</p><p><strong>From individual facilities to a connected research ecosystem</strong></p><p>As the NEMO community grows, another opportunity emerges. Universities and research organizations operate specialized facilities, each with unique equipment, expertise, and capabilities. But researchers cannot use a capability they are unaware of or cannot easily find.</p><p>This is particularly important for microelectronics R&amp;D, where progress often depends on moving quickly between specialized capabilities.&nbsp;A researcher may need fabrication at one facility, advanced characterization at another, specialized microscopy somewhere else, and a process or analysis capability available at only a handful of institutions.</p><p>Finding those capabilities, and knowing where and when they can be accessed, can itself become a bottleneck.Traditionally, networks of facilities have relied on static webpages or databases to describe available resources. But as facilities evolve, this information can quickly become outdated.</p><p>At the meeting, MIT.nano presented Hydra, a new open-source platform being developed with support from NEMC as a key component of the&nbsp;<a href="https://www.lab-hub.org/">NEMC Technical Concierge</a>, a program that aims to help researchers identify and access specialized capabilities across a network of facilities. Hydra is designed to connect multiple NEMO installations and draw information directly from the systems that facilities already use, rather than requiring someone to manually update a separate database. The result is a compilation built from the ground truths of actively-used facility management systems. “Hydra does not replace any lab management system,” Osherov explained. “It connects, synchronizes, and keeps data up to date across all connected systems in real time.”</p><p>In this model, NEMO provides the operational source of information within an individual facility, while Hydra makes that information discoverable across facilities.</p><p>For a researcher, that could mean finding a specialized capability without first needing to know which laboratory might have the relevant equipment.&nbsp;</p><p>“The focus is to make the distributed infrastructure supporting research more discoverabl

  • The result is a compilation built from the ground truths of actively-used facility management systems.
Nguyên văn nguồn
Chi tiết nguồn
Kỹ thuật

Learn the rules: The hacker mindset

Câu hỏi He also tackles an important question: are today’s networked systems any safer than they were a decade ago?

For some, rules pose limitations. For others, like Microsoft Research Manager Mike Walker, they’re opportunities. Walker’s fascination with rules and their role in the social order has allowed him to find success professionally and personally, from uncovering weaknesses in networked systems to redefining how computers are trained to fend off cyberattacks to pushing the limits of a reliable mode of transportation—the bicycle—in ultra-endurance racing. In this conversation with Principal Technical Program Manager Weishung Liu, Walker describes how his participation in hacker culture led him to a career building next-generation cyber defense, including his work developing the first all-machine hacking tournament. Along the way, he reflects on why effective security depends on understanding trust and context, what competitive hacking taught him about teamwork and collective intelligence, and how an early lesson reshaped his understanding of leadership. He also tackles an important question: are today’s networked systems any safer than they were a decade ago? Subscribe to the Microsoft Research Podcast : Apple Podcasts Email Android Spotify RSS Feed Transcript [MUSIC] MIKE WALKER: Well, you have to understand the rules if you want to be able to do what you want. So I have, I have a definite fascination with the rules, [LAUGHTER] but the thing you have to understand about the rules is that rules are a social construct. And I think that a lot of … in the computer security field, when, when people hear the word “security” and they think about the rules that the security system is imposing on them, every time a security system asks something of users, then you’re asking for their trust. And that trust is a precious commodity. And so when you squander user trust, then the security regime falls down. So I think it’s really important to understand the social context of rules instead of just having instant adjudication. [THEME MUSIC] STANDARD INTRODUCTION: This is the Microsoft Research Podcast , where Microsoft researchers—driving advancement through fundamental science and technology research—explore the who, how, and what’s next in computing and AI. [MUSIC ENDS] WEISHUNG LIU: Hi, and welcome. My name is Weishung Liu, and I’m a principal technical program manager here at Microsoft Research. And today, I’m joined by Mike Walker, who is a research manager here in MSR, or Microsoft Research, where he leads projects at the intersection of AI, cybersecurity, and large distributed systems. Before Microsoft, Mike led DARPA’s Cyber Grand Challenge, the pioneering program that demonstrated autonomous AI systems capable of finding and patching software vulnerabilities. And CGC actually just celebrated its 10th anniversary this year in 2026. Mike’s work spans groundbreaking efforts that include cloud native memory analysis, automated malware analysis, and next-generation cyber defense technologies. Mike, thanks for joining me. MIKE WALKER: Good to be here, Wei. LIU: All right, so, Mike, we have known each other for a couple of years now. And I’ve heard, like, bits and pieces of your story over time as we’ve gotten to talk. But before I really knew who you were, sort of, as a person, I, kind of, knew you as, like, “Mike Walker, the hacker guy that’s working over there,” which I’m sure you love. And I know you love that, when I call you “The Hacker.” But you’ve worked on so many things that I’ve never really thought about before. Like, new words have entered my vocabulary as a result of knowing you and working with you, like threat intelligence , nation states , and I think there are a couple other ones that I think would have never been in my lexicon had I not known you. So I’m hoping we can spend this time to also have the audience get to know you, as well. So we’re going to start at the beginning then. And the beginning is, I believe, Mike, you grew up in Ohio. WALKER: Columbus, Ohio. LIU: Columbus, Ohio. What kind of a kid were you in Columbus, Ohio? WALKER: I think I was a kid who spent a lot of time in the woods. [LAUGHTER] I was a skateboard kid. I was a basketball kid. But I think mostly I was a kid who read books and lived in a world of their own imagination. LIU: Were you on a farm in Columbus, Ohio, or were you in, like, or is this more suburban, rural, urban? WALKER: It was halves and halves. I spent my … most of the year in Columbus, Ohio, and then every summer, I would go out to the town of Tappahannock on the Rappahannock River in Virginia. And I lived completely on the farm. LIU: So what was the lifestyle in what you were doing different between, like, Columbus, Ohio, and the farm? WALKER: So, like, some things that I was doing in Columbus, Ohio, is that my father, who was sort of half scientist, half venture capitalist, would bring me home industrial equipment and say, “See what you can make out of this,” and drop it in my room. [LAUGHTER] Really dangerous stuff. And he would always give me safety warnings, right, like, “This is a helium neon laser.” You know, “This is … this tube is made out of glass. Don’t break it.” And then, “This is a really, really high-voltage power supply driving it, and it can arc and kill you,” right. And I knew what that meant. [LAUGHTER] And then he would leave me with it. And what can you make out of this, right. So I would do stuff like, like try to replicate the Michelson-Morley experiment, you know, because I had the, had the table and the mirrors and I could do that. And then also I spent a lot of time with the Apple II, which was this just incredible machine from some other universe that I had never gone to. I later found out that was, you know, hackerspace and, and information technology culture, but it came from nowhere and there was nobody I could talk to about it. And so I spent a lot of time trying to hook the Apple II up to stuff in my life and, and try to control it with that thing. LIU: OK, so you brought up some hacker stuff, so I have to ask. There w

  • Like, new words have entered my vocabulary as a result of knowing you and working with you, like threat intelligence , nation states , and I think there are a couple other ones that I think would have never been in my lexicon had I not known you.
Nguyên văn nguồn
Chi tiết nguồn
Tác tử

Agents SDK v0.24.0-alpha.6

<h1>Install the preview</h1> <p>Use standard CPython 3.10–3.14 in a virtual environment. These wheels include the native runtime; no Rust toolchain or source build is needed. This preview is distributed through GitHub Releases, not PyPI.</p> <p>Run the command for your platform:</p> <p><strong>Linux x86_64 (glibc 2.39 or newer)</strong></p> <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="python -m pip install &quot;https://github.com/openai/openai-agents-python/releases/download/v0.24.0-alpha.6/openai_agents-0.24.0a6-cp310-abi3-manylinux_2_39_x86_64.whl&quot;"><pre>python -m pip install <span class="pl-s"><span class="pl-pds">"</span>https://github.com/openai/openai-agents-python/releases/download/v0.24.0-alpha.6/openai_agents-0.24.0a6-cp310-abi3-manylinux_2_39_x86_64.whl<span class="pl-pds">"</span></span></pre></div> <p><strong>macOS 11 or newer on Apple Silicon</strong></p> <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="python -m pip install &quot;https://github.com/openai/openai-agents-python/releases/download/v0.24.0-alpha.6/openai_agents-0.24.0a6-cp310-abi3-macosx_11_0_arm64.whl&quot;"><pre>python -m pip install <span class="pl-s"><span class="pl-pds">"</span>https://github.com/openai/openai-agents-python/releases/download/v0.24.0-alpha.6/openai_agents-0.24.0a6-cp310-abi3-macosx_11_0_arm64.whl<span class="pl-pds">"</span></span></pre></div> <p><strong>Windows x64</strong></p> <div class="highlight highlight-source-powershell notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="python -m pip install &quot;https://github.com/openai/openai-agents-python/releases/download/v0.24.0-alpha.6/openai_agents-0.24.0a6-cp310-abi3-win_amd64.whl&quot;"><pre>python <span class="pl-k">-</span>m pip install <span class="pl-s"><span class="pl-pds">"</span>https://github.com/openai/openai-agents-python/releases/download/v0.24.0-alpha.6/openai_agents-0.24.0a6-cp310-abi3-win_amd64.whl<span class="pl-pds">"</span></span></pre></div> <p>Wheels are not provided for Intel Macs, ARM Linux, ARM Windows, Alpine/musl, PyPy or free-threaded Python. The <code>cp310-abi3</code> filenames also support standard CPython 3.11–3.14.</p> <p>Check the installed version and native module:</p> <div class="highlight highlight-source-shell notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="python -c &quot;from importlib.metadata import version; import agents._native; print(version('openai-agents'))&quot;"><pre>python -c <span class="pl-s"><span class="pl-pds">"</span>from importlib.metadata import version; import agents._native; print(version('openai-agents'))<span class="pl-pds">"</span></span></pre></div> <p>Expect <code>0.24.0-alpha.6</code> or its equivalent Python packaging spelling, <code>0.24.0a6</code>. <code>SHA256SUMS</code> contains checksums for all three wheels.</p> <h1>Release notes</h1> <p>Changes since <code>0.24.0-alpha.5</code>. This preview expands conversation control, state recovery, and tool policies, and includes public API changes.</p> <h3>Added</h3> <ul> <li>Control submitted work through a captured <code>Turn</code>, with <code>get_result()</code>, passive <code>events()</code>, manual request handling, steering, and cancellation. Queue prompts with <code>Session.submit(..., if_busy="queue")</code>, or steer the active Turn with <code>if_busy="steer"</code>.</li> <li>Add <code>LocalStorage</code> to reopen conversations from a local profile, and extend portable <code>SessionState</code> capture and recovery. Add idle forks, caller-supplied compaction summaries, and checks for feature support and checkpoint readiness. <code>LocalStorage</code> profiles cannot export portable checkpoints.</li> <li>Add <code>NativeOutput</code> for native strict output and <code>PromptedOutput</code> for JSON validated in Python. Optional <code>max_retries</code> performs bounded corrections without tools or replaying the task. <code>LocalStorage</code> requires <code>max_retries=0</code>.</li> <li>Handle live manual tool requests and resume supported pending requests from checkpoints. Add experimental external-job recovery through <code>agents.experimental.deferred</code> and an optional MCP Tasks adapter.</li> <li>Select tools for each submission or model step within the Session's opening permissions. Add native web search, native child checklists, and native goals for local-storage Sessions.</li> <li>Validate Python tool return annotations and add tool policies for inspecting arguments and results. Extend native MCP support with argument replacement, result transforms, and application-managed form and URL interactions over supported HTTP, stdio, and model WebSocket paths.</li> <li>Add <code>NetworkPolicy</code> for native commands on supported Linux and macOS executors, subject to each platform's network and Unix socket limits. Manage remote executor lifetimes through an <code>Environment</code> supplied when opening a Session or running an Agent.</li> <li>Enable native child approvals and form requests with <code>NativeAgents(interactions=True)</code>. Children continue to share the Session's tools and application context.</li> <li>Configure native OpenTelemetry export and inspect typed observations. <code>NativeError</code> now exposes available provider status codes, error codes, request IDs, and failure categories.</li> <li>Test real SDK sessions and Python tools against local scripted responses with <code>agents.testing.ScriptedResponses</code>, without a live model or API key.</li> </ul> <h3>API changes</h3> <ul> <li>Use <code>Agent.run()</code> or <code>Agent.run_sync()</code> for one-shot calls; the redundant <code>Runner</code> aliases have been removed.</li> <li>Replace <code>Turn.wait()</code> and <code>Session.wait()</code> with <code>Turn.get_result()</code> on the captured Turn.</li> <li

  • </li> <li>Validate Python tool return annotations and add tool policies for inspecting arguments and results.
Nguyên văn nguồn
Chi tiết nguồn
Tác tử

Show HN: An Open Source Superhuman Email Client Clone

Hi HN, I'm a long-time Superhuman user, and a few weeks ago I cloned it as a native Mac app. It's keyboard-first, with split inboxes, reminders, and a built-in calendar. It's local-first, including the AI features that use local models shipped in macOS 27, with the option to use Claude/Codex subscriptions. It's extremely configurable, even down to the search weights, and comes packed with CLI/MCP if you want to use it with an agent. This is my daily driver for email, but it's by no means a "stable" product. Use at your own risk. I'm also working on a mobile app to go along with it. Enjoy.

HN: open weights · grouchy 4 lượt bình chọn 0 bình luận Mở bản gốc ↗
Nguyên văn nguồn
Chi tiết nguồn
Đánh giá

GDPx: Measuring Frontier Model Capabilities Across the Most Valuable Domains

GDPx is a benchmark that covers the most valuable domains for frontier intelligence. It aims to be the world’s best benchmark for measuring general agentic capabilities. We believe the first trillion dollars of AI revenue will be concentrated in sectors that can deploy frontier intelligence at high ROI , not broadly diffused throughout the whole economy. Building a general-purpose benchmark based on share of GDP or the SOC/O*NET occupational taxonomy is therefore a mistake. If your goal is to track the progress of frontier models, you should focus on the domains where they will be most impactful. Specifically, we cover the following domains: Chip design Cybersecurity Biology Finance Legal Quant trading Recursive self-improvement (RSI) Hardware design Materials science Alignment Software engineering And intentionally forgo industries like government, real estate, and insurance. Interpreting the results GDPx is a true frontier-level benchmark: even the best model scores just 12%. Source: GDPx It’s worth emphasizing that certain models like Opus 5.5 and GPT-6 Astra have sandbagged overall scores due to their safety classifiers. If we exclude cyber, biology, and alignment—the three domains where safety classifiers sometimes kick in—you get the following results instead. Source: GDPx We were pleasantly surprised by how well this ranking lines up with our subjective impression of the models. OpenAI and Anthropic generally dominate the top of the ranking while SpaceXAI, Meta, and open-weight models are all meaningfully behind. Gemini 4 Carbon is a real improvement from Google and the only model in the same ballpark as Opus and Astra. It will be interesting to see people’s impressions once Carbon becomes generally available. We believe it’s underrated how relevant safety classifiers are when picking the best model for certain use cases. For example, Muse Wonder Full is actually the best available cyber model today by a meaningful margin . Source: GDPx We also generally observe meaningful spikiness across all leading models. Source: GDPx Opus 5.5 is the best all-around model and wins in quant trading, hardware design, and chip design. Gemini 4 Carbon is the best bio model (likely due to weaker safety classifiers) and edges out the win in software engineering. GPT-6 Astra wins convincingly for RSI and is also the best model for materials science. Our finance and legal tasks are so challenging that basically every model scores exactly 0. Interestingly, if you subset to a single model family, you will sometimes see very similar spikes. Source: GDPx The cost vs performance Pareto frontier also matches our general experience using the models: Luna and DeepSeek fight for the low end, Opus wins the high end, and OpenAI claims the middle. Again, however, the appearance of Gemini 4 Carbon is worth watching closely. Source: GDPx It’s impossible to make all the nuanced comparisons enabled by GDPx in a single newsletter. To explore the data yourself, checkout the official website . A collaboration with 14 specialized data partners Domain-specific benchmarks have gotten significantly better over the past year largely thanks to the rise of specialized RL environment startups. Yet OpenAI’s GDPval (along with its derivatives like GDPval AA v2.1 ) remains the most popular general purpose benchmark still reported on major new model release cards . This is despite being released over a year ago on September 25, 2025—an eternity in AI land. GDPval may have been marketed as a “knowledge work” benchmark, but agents weren’t really even a thing yet. This had a huge impact on its task design, and as we’ll later explain, makes GDPval woefully inadequate for evaluating today’s frontier models. Our goal with GDPx is to create the world’s best benchmark for general agentic capabilities. However, as soon as we started working on this project, it became clear why we’ve yet to see a worthy GDPval successor. Models like GPT-6 Astra and Opus 5.5 are already pushing the boundaries of human knowledge. If you want to make a finance or cyber task these models can only solve < 20% of the time, you need real domain expertise to ensure the task is hard for the right reasons. Many popular benchmarks sidestep this problem by shipping tasks that are downright unfair . This typically manifests in a grader that checks for requirements impossible to infer from the task description or environment. For example, a coding problem may ask the model to fix a bug that a human engineer would agree has two valid solutions—one using “module A” and another using “module B”. However, a poorly designed grader may require the AI to use “module A” and unfairly give 0 reward when the AI chooses “module B”. Correct all such issues, and the benchmark often ends up being saturated . Making high quality tasks for a single domain you have expertise in is already hard enough—there’s a reason why labs are willing to pay $5k+ for a single high quality task. Finding one company with sufficient expertise in both AI evals and enough domains to make a broad-based GDP benchmark is downright impossible. To solve this problem, SemiAnalysis partnered with specialized RL environment startups for each domain: Chip Design ( Aria , Arcverex , Ergodex ) Cybersecurity ( Arithmetic ) Biology ( Latch Bio ) Finance ( Living Context ) Legal ( Chakra ) Quant trading ( Third Moment ) RSI ( Extrapolate ) Hardware design ( Beautiful Machines ) Materials science ( ValenceForge , Emulated ) Alignment ( Fleet ) Software engineering ( Datacurve ) All partners were subject to a rigorous QA process we detail in the methodology section. For some domains, we had multiple partners who wanted to be exclusive and ran a process to select the one with the highest quality tasks. We would like to give a huge thank you to all of our partners for contributing to GDPx. They’re among the best vendors for their respective domains and it would’ve been impossible to create such a broad-based benchmark without

  • Specifically, we cover the following domains: Chip design Cybersecurity Biology Finance Legal Quant trading Recursive self-improvement (RSI) Hardware design Materials science Alignment Software engineering And intentionally forgo industries like government, real estate, and insurance.
  • Interpreting the results GDPx is a true frontier-level benchmark: even the best model scores just 12%.
  • If we exclude cyber, biology, and alignment—the three domains where safety classifiers sometimes kick in—you get the following results instead.
Nguyên văn nguồn
Chi tiết nguồn
Đánh giá

Q&A: Hey AI, who should I vote for this year?

Câu hỏi midterm elections, what role are AI platforms playing?

<p><em>People are peppering artificial intelligence chatbots everyday with questions, but when it comes to politics and the looming 2026 U.S. midterm elections, what role are AI platforms playing? Could an AI chatbot sway a voter to switch parties or re-examine their political beliefs?&nbsp;</em></p><p><em>MIT researchers have built a new dashboard, called the&nbsp;</em><a href="https://www.llmelectionobservatory.org/" target="_blank"><em>LLM Election Observatory</em></a><em>, to examine, in real-time, how popular AI chatbots are responding to questions about the 2026 midterm elections. The team behind the dashboard — led by&nbsp;</em><a href="https://polisci.mit.edu/people/adam-berinsky" target="_blank"><em>Professor Adam Berinsky</em></a><em>, </em><a href="https://mitsloan.mit.edu/faculty/directory/chara-podimata" target="_blank"><em>Assistant Professor Chara Podimata</em></a>,<em> and&nbsp;</em><a href="https://polisci.mit.edu/people/charles-stewart-iii" target="_blank"><em>Professor Charles Stewart III</em></a><em> — has extensive experience studying voting and election administration, AI and misinformation, and how AI systems are impacting human decision making. In an effort to better understand how AI is shaping the election, they designed the dashboard to simulate how individuals across the country communicate with chatbots.&nbsp;</em></p><p><em>The team tracks how different AI systems respond to the same fixed set of questions on a range of topics, including election certification, hot-button issues, result predictions, news coverage, and more. Every few days, the LLM Election Observatory team runs the questions through the chatbots again, monitoring how their responses vary as the election gets closer depending on user location, political party affiliation, identity and other factors. The goal of this work is not to merely understand how AI systems spread information, but also better comprehend how they are shaping political discourse and voter understanding of political issues across the country.&nbsp;</em></p><p>MIT News<em> chatted with Podimata, the&nbsp;Class of 1942 Career Development Professor,&nbsp;an assistant professor of operations research and statistics in the MIT Sloan School of Management, and a member of the Laboratory for Information and Decision Systems; and Berinsky, the Mitsui Professor and a professor in the Department of Political Science, about their work with the LLM Election Observatory, how AI could impact the midterms, and what voters should keep in mind when they interact with chatbots.</em></p><p><strong>Q</strong>: What was the inspiration for this project and what are you hoping to learn through the LLM Election Observatory?</p><p><strong>Podimata</strong>: People are increasingly using AI chatbots to find and make sense of political and election-related information. That makes it important to understand what information these systems actually provide, not just at one moment in time but throughout an election cycle. This is particularly challenging because the models themselves are continually getting updates, the political information inputs available to them are themselves changing, and their answers can depend on how a user asks a question or what information the user provides about themselves.</p><p>One basic problem is that we currently do not have a good historical record of this information environment. If an AI system gives a particular answer about an election today and gives a different answer several weeks later, it can be very difficult to reconstruct after the fact what voters would have seen at each point in time. One important goal of our project is simply to systematically document that environment as it develops and make the resulting data available to other researchers and to the public.</p><p>Ultimately, factual accuracy is only the first frontier. We absolutely need to understand whether chatbots provide accurate political information, but we also need to pay just as much attention to the subtler ways in which they may shape people’s understanding of political issues. Differences in framing, emphasis, uncertainty, and cited sources can matter enormously, even in the absence of outright misinformation.&nbsp;&nbsp;</p><p><strong>Q</strong>: Professor Podimata, you studied how LLMs shaped political discourse and information during the 2024 election. What were the main takeaways of that research and how did those findings inform this new project?</p><p><strong>Podimata</strong>: The LLM Election Observatory builds on my previous&nbsp;<a href="https://arxiv.org/abs/2509.18446" target="_blank">work</a> during the 2024 presidential election, which found evidence of meaningful variation in AI models’ responses over time and across prompt framings, as well as differences in how models characterized candidates and implicitly anticipated the election outcome.&nbsp;</p><p>The 2026 midterms give us an opportunity to expand that work substantially. Midterm elections are particularly interesting because there are many state and local contests unfolding simultaneously, alongside broader national political narratives. Our new audit survey therefore asks both contest-specific questions about the U.S. House and Senate candidates, but also broader questions about voting information and other aspects of the election information environment.</p><p><strong>Q</strong>: Why do you think it’s so important to be studying how chatbots are responding to political queries across different groups of users?</p><p><strong>Podimata</strong>: We all build our own “microcosms” through our interactions with chatbots. And yet, we lack systematic ways of understanding how these individualized information environments are shaped by who we are or how they evolve over time as we and the information environment changes.&nbsp;</p><p>For example, people may not realize that what their favorite chatbot tells them about a political issue may be quite different from what the

  • &nbsp;</em></p><p><em>The team tracks how different AI systems respond to the same fixed set of questions on a range of topics, including election certification, hot-button issues, result predictions, news coverage, and more.
Nguyên văn nguồn
Chi tiết nguồn
Đánh giá

MInfoVisQA: A Multilingual Benchmark for Information-Rich Visual Question Answering

Multimodal large language models (MLLMs) excel in vision-language tasks; yet, whether they maintain consistent understanding of information-rich visuals (InfoVis) when the surface language changes remains unclear. To quantify this capability, we introduce MInfoVisQA, a parallel benchmark with 8,960 QA configurations derived from 128 seed instances across 24 languages. We construct seed-matched, language-parallel visuals and QA pairs by disentangling numerical-structural data from text and programmatically synchronizing translated labels. We evaluate 25 MLLMs under two settings: Language-Aligned QA (LQA, matching visual-text languages) and Cross-Language QA (XQA, fixed QA language with varying visual languages). The results show that aggregate accuracy masks substantial robustness gaps: two models with comparable mean LQA accuracy (80.4\%) exhibit performance spreads of 7.8 and 20.3 percentage points across languages. Moreover, all 25 models score lower with non-English visuals under fixed English QA, underscoring the need for cross-language sensitivity metrics.

Hugging Face Papers · Changjiang Jiang, Jiang Zhou, Xinbin Yuan, Fan Zhang, Tao Feng, Zhuohan Xie 1 lượt bình chọn 1 bình luận Mở bản gốc ↗
Nguyên văn nguồn
Chi tiết nguồn
Huấn luyện

Microsoft Research Focus – Edition #97

Câu hỏi But how can we count on AI agents to do the work if their ability to adjust when things go wrong may require a lot of resources with no guarantees of a better result?

What’s new in research? You may already be familiar with the Research Focus newsletter (opens in new tab) on LinkedIn. Now, you can catch up on highlights from across the research community at Microsoft on our social channels, YouTube, and major podcast platforms, too. In this edition of Microsoft Research Focus , host Sowbhagya Suresh shares some of the latest news and cutting-edge research from across Microsoft—from understanding the challenges facing coding agents to uncovering how people use AI at work and improving extended-range weather forecasting. Catch up on the ideas, tools, and discoveries researchers are exploring today and get a quick look at what’s coming next. Related links: Research Focus newsletter on LinkedIn (opens in new tab) ForkPilot: Self-Evolving Policy for Retrospective Search in Long-Horizon Agents Code Understanding is a Bottleneck for Coding Agents TeleTune: Evolving Agent Skills from Offline Telemetry Why Copilot use looks so similar across industries, and what that misses (opens in new tab) Shiksha Copilot: Teacher-AI Collaboration for Curating and Customizing Lesson Plans in Low-Resource Schools New AI framework bridges a critical gap in mid-range weather forecasts BioEmu2 in public preview (opens in new tab) Chapters 00:48: A self-improving agentic framework (opens in new tab) 01:35: Understanding coding agent success and failure (opens in new tab) 02:29: Evolving agent skills with activity logs (opens in new tab) 03:22: Looking beyond similarities in AI use (opens in new tab) 04:15: Helping teachers make more time for teaching (opens in new tab) 04:52: Improving extended-range weather forecasting (opens in new tab) 05:42: BioEmu2 now available in public preview (opens in new tab) 06:32: Final thoughts (opens in new tab) Subscribe to the Microsoft Research Podcast : Apple Podcasts Email Android Spotify RSS Feed Transcript [MUSIC] SOWBHAGYA SURESH: Hi, I’m Sowbhagya Suresh. Welcome to Microsoft Research Focus . You’ve probably read our newsletter. And now you can get highlights from across the research community at Microsoft here, on YouTube, and major podcast platforms. Whether you read, watch, or listen to it, Research Focus is your source for research news at Microsoft. Coming up in this edition: what everyday AI use reveals about the way we work; how a new AI model could help scientists better understand proteins; a new approach to improving weather forecasts weeks in advance; and more. So here’s what’s new. A self-improving agentic framework More and more, AI agents are handling tasks that are complex. [MUSIC FADES] Tasks that require multiple steps, decisions, and interactions with external tools. But how can we count on AI agents to do the work if their ability to adjust when things go wrong may require a lot of resources with no guarantees of a better result? A new self-improving framework called ForkPilot learns when it’s worth revisiting earlier decisions to find a different approach and adapts its strategy based on new outcomes. Tested across six benchmarks and leading AI models, ForkPilot measures up to the best existing methods while reducing AI processing demands by up to 59%. These results suggest that AI agents can solve complex problems more reliably and efficiently, making them more practical and cost-effective to deploy at scale. [TRANSITION MUSIC] Understanding coding agent success and failure Coding agents execute user queries by reasoning about, editing, and running code. Understanding why these agents fail is critical if we want to build more capable and reliable coding agents. Unfortunately, existing benchmarks often make it difficult to identify which factors contribute to success or failure. CABRA is a framework for generating coding challenges from scratch. It lets researchers increase complexity in a controlled way and pinpoints the skills that are most difficult for agents. Across thousands of generated challenges, the researchers found that agents often overcame coding problems by using tools. Yet the agents still struggled when they needed to understand unfamiliar codebases. The findings suggest that understanding existing code, rather than modifying it, remains one of the biggest obstacles to developing more effective coding agents. [TRANSITION MUSIC] Evolving agent skills with activity logs Computer-use agents are AI systems that are designed to complete tasks that require clicking and typing. They can fill in forms, process requests, and navigate websites. To do this effectively, they need to learn the procedures that people use to get work done. User activity logs can offer a way to capture this knowledge. But activity logs typically don’t record the intent behind each action and often combine tasks. The new framework TeleTune learns reusable skills and workflows from these logs. It also infers the task goals associated with the sequences of actions. In evaluations across software and browser tasks, TeleTune outperformed existing approaches. By learning from existing activity, TeleTune offers a scalable way to help AI agents adapt to unfamiliar software and carry out complex, multi-step workflows [TRANSITION MUSIC] Looking beyond similarities in AI use (opens in new tab) Workers across industries may appear to be using AI in surprisingly similar ways, but a closer look reveals a different story. Microsoft researchers analyzed aggregated Copilot Chat activity across a broad range of customer categories and found that editing documents led all classified activities, accounting for more than 20 percent. But when researchers set aside the 20 most common activities, differences between customer categories became substantially more pronounced. The most common uses of AI may be hiding important differences in how organizations use it for more specialized work. For business leaders, the takeaway is clear: don’t assume you have the same AI needs as others in your industry. Finding the right opportunities means looking beyond usage

  • In this edition of Microsoft Research Focus , host Sowbhagya Suresh shares some of the latest news and cutting-edge research from across Microsoft—from understanding the challenges facing coding agents to uncovering how people use AI at work and improving extended-range weather forecasting.
  • Coming up in this edition: what everyday AI use reveals about the way we work; how a new AI model could help scientists better understand proteins; a new approach to improving weather forecasts weeks in advance; and more.
  • A self-improving agentic framework More and more, AI agents are handling tasks that are complex.
Nguyên văn nguồn
Chi tiết nguồn
Đánh giá

Artificial Analysis v5 and the benchmarker's dilemma

Hill-climbing is easy, knowing what to hill-climb is hard. Making scores go up is expensive , but we have a much clearer playbook for it than we do for creating meaningful benchmarks at the true frontier of AI today. Independent evaluation of language models is under-appreciated in how hard of a task it is. These groups are not just taking current models and certifying their scores, which is a technically involved task. This is their job if AI progress is slow. When AI progress is as fast as it is, the technical details of the job are the routine work part of trying to put a single number on what makes for a good AI model today. The goal is to have a single number that is representative of current models, matching vibe checks, community sentiment, and scientific analysis, while being forward looking enough so that it’s not outdated in 3 months. Everyone building a benchmark faces a sort of paradox. You need your benchmark to reduce to one single number, so that it is easy to discuss, use in reports, and compare models, but every method of aggregating scores comes with meaningful trade-offs. The best scientists need to do this, everyone does. It’s a matter of creating aggregation metrics that you think are most fair, knowing there will always be imperfections. With the amount of money in the AI ecosystem today, the weight of independent evaluation is even higher – you dictate the winners and losers of a trillion dollar market. You also influence some of the models themselves, steering the shape of progress. This is a lot of pressure! My prompt for this article is Artificial Analysis’s recent update to their Intelligence Index, v5 . The AA index is the most prominent independent intelligence metric for the latest frontier models, followed in a distant second by Epoch AI’s Capabilities Index (ECI) . To date, all of these have been incredibly useful for the community and massively flawed! Interconnects AI is a reader-supported publication. Consider becoming a subscriber. Language models are capable of so much today, and this scope of capabilities is growing quickly. Our era of singular intelligence indices should be phased out soon, but it will take a long time. A summary of AAv5 is below, from their blog post, where they describe it as “reflecting real-world usage today:” Intelligence Index v5 weights: ➤ Knowledge Work (25%): AA-Briefcase (15%), AutomationBench-AA (5%), and GDP.pdf (5%) ➤ Coding (25%): FrontierCode 2.0 (15%) and Terminal-Bench 4.0 (10%) ➤ Science (25%): Terminal-Bench-Science (15%) and CritPt v2 hard subset (10%) ➤ General (25%): AA-Omniscience v1.1 15% (10% accuracy, 5% non-hallucination) and HLE-Diamond (10%) Here’s the new ranking of models: The problem with trying to benchmark real-world usage today is that it is extremely jagged, just like the models. The most proven use-case is coding, where open models are far more competitive and likely to keep up. In knowledge work, I’m less confident in open models. At first glance, in aggregate, their model ranking definitely looks reasonable. Still, it shows signs of strain and confusion, as any benchmark will, with GPT 6.1 Terra beating Claude Fable 5.1 on AAv5. Today, some benchmarks measure the absolute frontier of capabilities, playing nicely in open-ended tasks through your favorite harness, which often does take a large model. Some other benchmarks measure certain skills that will be needed of a model in repetitive agentic tasks. I get the feeling that AA is tuning their benchmark more towards the former, a general agentic collaborator, but all of these behaviors overlap. This takes us to the plot going around that prompted me to write this, where they showed the balance of open vs. closed models on the new benchmark. This plot shows 8 closed models beating the best to-be open-weight model, Kimi K3.1 Pro (which isn’t colored blue). This benchmark is taking the opinion of trying to measure where frontier models are moving into, which grows the gap to open models. This is an okay decision to take, but over time as existing workflows of open models become key factors of the economy, it could become a benchmark that misses where the core economic engine of the industry is. Open models’ demise from this seems greatly exaggerated, but I do think the previous index was flattering the open weight models too strongly. Consider this plot from the Kimi K3.1 Pro launch — yes, the open models are strong and useful in some cases, but the best OpenAI and Anthropic models have been a major step ahead in their flexibility and reliability. Some of the evaluations in AAv5 that open models are most behind on are “science” evals like terminal-bench-science and CritPT v2 . Open models are behind, but the value proposition was never to compete truly at the frontier. The open model economy will be about Reducing domains with high market demand for inference to a commodity price of hardware. Allowing ownership of intelligence, specializing it to internal data, and controlling internal agents across many companies in the economy. Both of these will take a very long time to materialize and are partially represented in benchmarks. The supply of compute is currently so low that open model providers also are raking in solid margins – when this shrinks, we will know the system is starting to work. The same goes for diffusion into enterprises, it’s not clear what the exact ceiling on it is because of the difficult ML expertise that is needed to build evaluations and do post-training today. Share At the next major model cycle, say in 2-3 months with GLM 6 or Kimi 3.5, I expect the leading open model labs to make substantial steps on this index, again surprising many observers. This is a cycle we have been on for years now. I’ve written many times about how open models keep up with the frontier, and a summary is that: The best open models do benchmax a little. The worst open models benchmax a lot (but don’t get a ton of attention anyways). The best open mo

  • You need your benchmark to reduce to one single number, so that it is easy to discuss, use in reports, and compare models, but every method of aggregating scores comes with meaningful trade-offs.
  • The open model economy will be about Reducing domains with high market demand for inference to a commodity price of hardware.
Nguyên văn nguồn
Chi tiết nguồn
181–200 / 738
Trang 10 / 37