AllyshipContent Radar

Hands-on 193

Công cụ, thử nghiệm và hướng dẫn thực hành

Crawl live-1795068900000 · 13:22 19 thg 11, 2026 · Xuất bản 13:29 19 thg 11, 2026

73

What Frontier Labs Buy: A Data Deep Dive with Handshake AI — Garrett Lord & Team

40.9

Hướng dẫn swyx .ai

This is our first Latent Space deep dive with an AI data company . In roughly a year, Handshake’s AI training business went from launch to $1 billion in gross annualized revenue . What are frontier labs buying at that scale? What evals are models still nowhere close to human level on? Garrett Lord brought the people who build these reinforcement learning environments to show us. For the first time, we are walking through them live! To cover the scope of Handshake, we had to produce a very special joint podcast across our Science/Engineering/Research/Finance podcast hosts, because Handshake’s business runs across all these and more. Watch for three segments after the general intro: Finance. In the finance demo, the agent has to find its assignment in a bank’s email, work out which files govern the deal, and deliver a workbook. Watch what happens to one line in the client’s email about existing goodwill. The agent carries the number into Python and subtracts it. Collin points out that the model can explain the tax treatment when you ask it directly. The mistake comes when it has to recognize that this particular line of code is where the rule applies. Bio. Nick Magazine and Jonas Mueller show us VIALS, Handshake’s benchmark for interpreting the images scientists use at work. Nick puts nine slides of brain-tumor tissue on screen and asks which two show the most advanced vessel necrosis. Give yourself a moment before his answer. Nick picks H and I. The model picks those two and adds G. Watching Nick point to what’s missing from the tissue is more instructive than seeing the score. Coding. In a simulated file-access task, the team can inspect which files an agent read, whether it checked the copies, and what it did to the shell history afterward. A transcript of the agent’s final reply would miss most of what Steve Kelling and Nishant Ranka are evaluating. Who’s in the room 🟣 AI data business — Garrett Lord, Sahil Bhaiwala and Nishant Ranka open and close the episode on the market for expert data. 🌸 Life sciences — Nick Magazine and Jonas Mueller put the gel and brain-tissue tasks on screen. 🟢 Finance — Collin Schweiker and Sahil Bhaiwala take us through Project Falcon and its Excel grader. 🟠 Alignment — Steve Kelling and Nishant Ranka follow the agent’s tool calls through the safety simulations. Show notes ATLAS-Finance — Explore the benchmark for agents working across a simulated bank’s inboxes, documents, and financial models. Inside a bank — Handshake traces a separate published deal, Project Osprey, from the client’s email to a grading decision. The deal on screen in our interview is Project Falcon. VIALS sample tasks — Try the scientific image questions, including the gel and brain-tissue tasks from the episode. How Handshake trained a model to read scientific figures — The team describes using intermediate image descriptions to train and evaluate a vision-language model. Timestamps 0:00 Inside Handshake AI 0:39 Garrett, Sahil and Nishant on frontier training data 5:15 How to build a banking environment for an agent 13:07 How frontier labs buy data 15:51 How a lab evaluates a new dataset 20:01 Testing training data on open models 27:38 Where rubrics fall short of expert judgment 31:30 Nick and Jonas introduce the VIALS benchmark 35:52 Reading molecular structures and plasmid maps 43:48 Training vision models to interpret scientific images 47:17 Task one: reading the bands on a gel 50:59 Task two: finding necrosis in brain tissue 55:49 Can Python tools help the model see? 1:02:52 Can biology models improve on their own? 1:06:19 What a biology lab can buy from Handshake 1:12:33 Atlas Finance with Collin and Sahil 1:18:56 Inside a simulated investment bank 1:25:45 Where the agent’s financial model went wrong 1:31:18 How a banker grades the spreadsheet 1:41:12 Penalty traps, dependency gates and partial credit 1:46:41 How an agent gamed the Excel grader 1:48:52 What would make an AI banker useful? 1:54:48 Safety and alignment with Steve and Nishant 1:58:25 CAREBench and the limits of a refusal 2:04:42 How safety feedback reaches model training 2:07:21 When an agent sends the wrong files 2:11:14 A loan application tests fair lending 2:16:09 Designing private benchmarks for enterprises 2:25:37 The leadership team on Handshake’s mission 2:28:36 When enterprises need their own models 2:33:03 The market for compute and expert data 2:39:12 Why labs hold out evaluation tasks 2:43:01 Hiring at Handshake AI

Latent Space · 10 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
74

What Frontier Labs Buy: A Data Deep Dive with Handshake AI — Garrett Lord & Team

40.8

Hướng dẫn Latent Space

Our first deep dive with an AI data company. Handshake AI went from launch to roughly $1 billion in annualized gross revenue in about a year. What are frontier labs actually buying? Garrett Lord and the team open up the reinforcement learning environments behind their expert training data, from biology and investment banking to alignment. An AI agent found a way to earn credit for net income: open a new Excel tab and fill it with every number from one to a million. The grader found the number it wanted. A banker would have looked for it in the right row, for the right year. Garrett Lord and the Handshake AI team take Latent Space inside the work of building evaluations for frontier labs. A scientist walks us through a gel and nine brain-tissue slides. A banker follows an agent through its inbox, Python code, and financial model to find the assumption that broke the deal. The safety team shows how to read an agent’s actions when its final answer sounds fine. Opening and closing Garrett Lord, Sahil Bhaiwala, and Nishant Ranka Life sciences Nick Magazine and Jonas Mueller Finance Collin Schweiker and Sahil Bhaiwala Safety and alignment Steve Kelling and Nishant Ranka Explore the benchmarks • Try the VIALS visual life sciences tasks: https://joinhandshake.com/research/benchmarks/vials/ • Explore the ATLAS Finance banking environment: https://joinhandshake.com/research/benchmarks/atlas-finance/ • Read Handshake AI’s research: https://joinhandshake.com/research/ai/ Chapters 00:00 Inside Handshake AI 0:39 Garrett, Sahil and Nishant on frontier training data 5:15 How to build a banking environment for an agent 13:07 How frontier labs buy data 15:51 How a lab evaluates a new dataset 20:01 Testing training data on open models 27:38 Where rubrics fall short of expert judgment 31:30 Nick and Jonas introduce the VIALS benchmark 35:52 Reading molecular structures and plasmid maps 43:48 Training vision models to interpret scientific images 47:17 Task one: reading the bands on a gel 50:59 Task two: finding necrosis in brain tissue 55:49 Can Python tools help the model see? 1:02:52 Can biology models improve on their own? 1:06:19 What a biology lab can buy from Handshake 1:12:33 Atlas Finance with Collin and Sahil 1:18:56 Inside a simulated investment bank 1:25:45 Where the agent’s financial model went wrong 1:31:18 How a banker grades the spreadsheet 1:41:12 Penalty traps, dependency gates and partial credit 1:46:41 How an agent gamed the Excel grader 1:48:52 What would make an AI banker useful? 1:54:48 Safety and alignment with Steve and Nishant 1:58:25 CAREBench and the limits of a refusal 2:04:42 How safety feedback reaches model training 2:07:21 When an agent sends the wrong files 2:11:14 A loan application tests fair lending 2:16:09 Designing private benchmarks for enterprises 2:25:37 The leadership team on Handshake’s mission 2:28:36 When enterprises need their own models 2:33:03 The market for compute and expert data 2:39:12 Why labs hold out evaluation tasks 2:43:01 Hiring at Handshake AI

Latent Space · 9 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
75

In this session from Interrupt London, @hwchase17 discusses the "North Star" vision of building AI agents that improve over time. He defines an agent as a system consisting of three core parts:

40.5

Hướng dẫn vitalflux

In this session from Interrupt London, @hwchase17 discusses the "North Star" vision of building AI agents that improve over time. He defines an agent as a system consisting of three core parts: 1. The Model (the reasoning engine), 2. The Harness (the surrounding control system) 3. The Context (memory and knowledge). Key dimensions of agent improvement: * Model: Improved by updating weights through Supervised Fine-Tuning (SFT) for simpler tasks or Reinforcement Learning (RL) for complex tasks. * Harness: Involves refining the code, prompts, and tools. LangSmith Engine helps automate this by monitoring traces to identify issues and propose fixes. * Context: The easiest dimension to update; involves evolving memory at the user or organizational level. Foundations for improvement: * Traces and Evals: Harrison argues these are the critical foundation for all improvement. He highlights the need for specialized tools like SmithDB and LangSmith AI Search to handle the unique nature of agent trace data. * Evals as the new PRD: Evals are essential for defining "good" behavior, though they are difficult to build. He covers the challenges of creating realistic inputs, defining correctness, and running simulations without impacting production systems. * Red Teaming: Proactively identifies agent mistakes before they reach production, using algorithmic probing rather than relying solely on user feedback. The Future: * The ultimate vision is a "flywheel" where an agent can notice its own mistakes, create its own evaluation data, and suggest its own fixes to its model, harness, or context. #AgenticAI #AIAgents How to Build Agents That Improve Over Time: Model, Harness, Context https://youtu.be/xy_7SzwsIjM

X · 9 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
76

AI-assisted learning platform helps robots reason beyond literal commands

39.9

Hướng dẫn Jennifer Chu | MIT News

<p>Humans are intuitive by nature. We can sense meaning beyond the literal words that we encounter. Now, a new artificial intelligence-assisted learning platform developed by MIT engineers is enabling robots to read larger meaning into a human’s natural language commands.&nbsp;</p><p>In a study to be presented at the annual Conference on Robot Learning this month, researchers from MIT and New York University introduce <a href="https://arxiv.org/pdf/2610.28893" target="_blank">FLEX</a>, a flexible feature-editing pipeline for robot motion-planning.&nbsp;</p><p>With FLEX, humans can interact with a robot by giving it instructions in natural, conversational language. The robot can not only follow the instructions literally but also read into the words to draw larger generalizations that help to improve its motion-planning task.&nbsp;</p><p>The team demonstrated FLEX in several simulated scenarios, including one in which a robot serves a cup of coffee. In these virtual scenarios, a person interacts with a robotic arm by typing commands in natural language. As the robot moves the coffee cup over a cluttered table, the person might type “Don’t spill on the laptop!” In response, the robot will reason about why it should avoid the laptop. For this, FLEX incorporates a large language model (LLM) — a type of AI system that is trained on enormous libraries of text, including social media posts, video transcriptions, and web content, which the LLM uses to identify larger trends, patterns, and generalizations.&nbsp;</p><p>“LLMs have seen so many texts from the internet,” says study author Nathan Dennler, a postdoc in MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL). “People might have written about, ‘Oh, my phone dropped in the water and now it doesn’t work,’ or ‘I spilled coffee on my computer and now it doesn’t work.’ So LLMs pick up on the idea that liquids plus devices leads to things not working.”</p><p>Through the LLM, the robot reasons that it should also avoid bumping into all electronics. In its updated path, the robot adjusts to avoid both the laptop and a nearby phone, without the person having to explicitly spell this out.</p><p>“The idea is that there is some higher-level intention behind the words that you say,” Dennler says. “People naturally get this. But you have to carefully explain that to a robot, which takes time and effort. FLEX gives you an interface of communication with a robot to say, ‘When people say this, they mean that.’”</p><p>The study was led by Andreea Bobu, assistant professor of aeronautics and astronautics at MIT. Co-authors are Audrey Lee SM ’26 and Ilia Sucholutsky of New York University.</p><p><strong>Beyond what we say</strong></p><p>At MIT, Bobu heads up the Collaborative Learning and Autonomy Research (CLEAR) Lab, where her group develops autonomous agents that learn to work for, with, and around people. She is focused on developing assistive and household robots that can carry out physical tasks, and reason and adapt in real-time, in response to a person’s natural, conversational commands.&nbsp;</p><p>“Robots use math to do their trajectory planning. But we can’t expect a consumer to learn to do this math in order to tell a robot what to do,” says Dennler, who is a member of the CLEAR Lab. “Instead of having to get people to learn how to use a robot in a specific way, the goal is to interact with the robot through natural language.”</p><p>There is a large effort in the robotics community to incorporate natural language into robotic learning pipelines. Various strategies include using language to clarify a task’s goal, redirect a robot’s trajectory, and update its understanding of what is a safe path to take. These existing methods use natural language to tell the robot how to physically perform a task. But a robot could potentially learn much more from what we tell it.&nbsp;</p><p>Bobu and her colleagues are expanding what robots can learn from natural language commands. In their new study, they focused on enabling a robot to interpret a person’s underlying, unspecified preference from limited natural language commands.&nbsp;&nbsp;</p><p><strong>Absorbing common sense</strong></p><p>The team’s new robot learning platform, FLEX, is a three-step process that helps a robot read more into a natural language command, beyond what it is explicitly told to do.&nbsp;</p><p>The first step, “Add,” instructs a robot to add a feature to its list of things in its environment that matter for the task at hand, based on a user’s natural language command. For instance, as a robot is moving a bowling ball from one location to another, a person might interject with “Don’t drop it on my mug!” The robot would then add the mug to a list of things to avoid in its path. This first step is how previous methods have worked. Every time a person makes a correction, the robot adds to its features list, and updates its trajectory to avoid those features.&nbsp;</p><p>“The problem happens when your feature list becomes super long, and some features are no longer relevant because your environment may have changed,” says Dennler, who notes that this approach can grow tiresome if a person has to specify every specific feature to avoid.&nbsp;</p><p>The second step, “Generalize,” is the new solution that FLEX provides. This step incorporates an LLM — in this case, GPT-4o — that considers the mug, along with any other features that the person has so far specified to avoid. The LLM automatically searches through its vast cache of data for patterns and connections among these features, to ultimately make a generalization that it should avoid objects that are fragile, even when the person has not explicitly said so.&nbsp;</p><p>“LLMs have&nbsp;seen so many texts from the internet, from all sorts of people and places,” Dennler says. “Because they’ve trained so much on those things, some of that common sense gets absorbed into the LLM.”</p><p>In the third step, “Mask,” the robot’s

MIT News: AI · 9 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
77

How Stanford Teaches AI-Native Engineering — Mihail Eric, Monaco

39.3

Hướng dẫn swyx .ai

The AI Engineer YouTube channel just crossed one million subscribers . Thank you to everyone who’s watched a talk, sent one to a teammate, or put something they learned into production. For this special, we wanted to sit down with someone who has to answer a question a lot of us are asking at work: what should a software engineer actually learn now? Mihail Eric has two places to test his answer. He teaches The Modern Software Developer at Stanford and leads AI at Monaco, where his colleagues use coding agents to ship software and investigate outages. I asked him to take us through the syllabus, then tell us which parts survived contact with his own engineering team. This is also meant as a preview for all of you joining us in San Francisco for AIE Code next week, as required knowledge for all the conversations with the speakers, sponsors and attendees you’re about to have. If you can see this as a LS subscriber and haven’t got a ticket yet, you can use this code for 30% off (no refunds, sorry) Last year’s homework takes one prompt The first run of CS146S had nine assignments and a capstone. In one exercise, students ran an AI code reviewer on a repo and evaluated its feedback. By the next run, Mihail figured students could one-shot a lot of those exercises. When he asked a room of Stanford sophomores and juniors how they coded, everyone he surveyed was already using an agent. So this time, part of the homework is contributing to an open-source project. The person who maintains the repo reviews the student’s patch. I like what that does to the assignment. You can ask an agent to implement a feature in an afternoon. You still have to find out whether the maintainer wants it, why the code behaves the way it does, and what else your patch might break. That conversation is also how a junior engineer gets to learn from a senior one. There’s an exercise in the episode you can borrow for your next PR. Geoffrey Litt’s explain-diff skill turns a change into a visual walkthrough, then quizzes you about it. Try it on a patch your agent just wrote. Which question do you have to reopen the code to answer? Watch the discussion . What Monaco buys, and what it builds At Monaco, the team evaluated code review products and kept using Devin. Agent workspaces went the other way. A background agent needs somewhere to run the product, reproduce a failure and test the patch. The details depend on how Monaco’s cloud is set up, so the team built its own workspaces. Mihail’s rule of thumb: “You buy to learn, and you build to keep.” One result is Hal. An engineer can ask it to investigate something that broke and open a PR. A salesperson can ask the same system how the product works. In one case the useful answer is a change someone can review. In the other, it’s an explanation you can give a customer. Give both people a wall of debugging output and you’ve only served one of them. The question I kept coming back to was what happens at the other end of the PR. An agent sees the current code. Where does it look to find out why the team rejected this same change six months ago? That takes us from Basis’s Authority Map to Mihail’s prediction that open-source repos will become agent-run software factories. It’s a good reason to send a student to a maintainer today. Someone still has to decide which change belongs in the project. If you’re building with coding agents, come join us at AI Engineer CODE . And if you’ve been teaching this stuff yourself, I’d love to hear which assignment you had to rewrite first. Show notes Stanford CS146S — The ten-week syllabus, assignments and open-source partners. You can also switch to the previous year’s course to compare. Build a coding agent and look inside Claude Code — Mihail’s walkthroughs of the tool loop and the harness around it. Understanding is the new bottleneck — Geoffrey Litt’s approach to learning what changed in an AI-written diff. Monaco’s agent-developer workspaces — The team’s reasoning for building a place where agents can run and test the app. How to Kill the Code Review and the underlying Faros study — Teams with high AI adoption merged 98% more PRs, while their PR review time was 91% longer. Basis’s Authority Map — Where an agent should look for current behavior, earlier decisions and intent. Stripe’s Minions — A production example of giving background coding agents an environment and a path to an engineer-reviewed PR. Boris Cherny’s Steps of AI Adoption — The five stages we pull up on screen near the end of the episode. Mihail on The Thinking Mode — EO Studio’s interview, excerpted at the start of ours. Timestamps 0:00 · One million AI Engineer subscribers and the CODE Summit 0:51 · Mihail Eric and Stanford CS146S 5:26 · What junior engineers need to learn now 10:02 · Open-source maintainers review the students’ code 15:04 · A tour of the ten-week syllabus 24:04 · The loop, tools and harness inside a coding agent 32:18 · Context engineering for a large codebase 38:15 · Skills that make you understand the diff 46:42 · The Monaco.com dispute and the airplane banner 50:23 · What Monaco does for sales teams 59:21 · Inside Monaco’s engineering team 1:01:40 · “You buy to learn, and you build to keep” 1:05:24 · Making repos agent-ready and closing the on-call loop 1:12:37 · When code review becomes the bottleneck 1:20:37 · Security, dependencies and permissions 1:30:13 · Background agents and Stripe’s Minions 1:34:16 · MCP portals and why infrastructure converges 1:41:22 · Building the software factory 1:45:24 · What work is left for developers? 1:52:16 · From personal adoption to team autonomy Transcript Download the complete text below. In the web edition, you can also read the transcript and click any timestamp to jump into the interview. transcript.txt 152.95 KB • PLAIN File Automated transcript, aligned to the final episode. Timestamps open the corresponding moment on YouTube. 0:00 swyx · Intro — Okay, this is a pretty special episode because we're celebrating 1 mi

Latent Space · 9 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
78

[One Million AI Engineer Special] How Stanford Teaches AI-Native Engineering — Mihail Eric, Monaco

39.3

Hướng dẫn Latent.Space

What do you assign when a coding agent can finish last year’s homework in one prompt? In the second run of Stanford’s CS146S, Mihail Eric brought open-source maintainers into the course. Students have to learn an unfamiliar repo, propose a change, and explain the patch to the person who might merge it. We invited Mihail, who teaches The Modern Software Developer at Stanford and leads AI at Monaco, to celebrate one million subscribers on the AI Engineer YouTube channel. He opens up the course syllabus and shows us which lessons survived contact with his own engineering team. Monaco built its agent workspaces, bought its code review tools, and gave an on-call agent access to Datadog and GitHub. We ask how he decides which tools his team needs to own. Watch the full interview and screenshares on YouTube . Show notes Stanford CS146S — The ten-week syllabus and assignments. The Emperor Has No Clothes — Mihail builds a coding agent from scratch. Demystifying Claude Code — Mihail traces a real agent harness. Monaco’s agent workspaces — How the team sets up a workspace for each agent. How to Kill the Code Review — Our essay on what happens when code generation outruns review. Basis’s Authority Map — How agents distinguish current behavior from earlier design decisions. Stripe’s Minions — A look inside Stripe’s background coding agents. AI Engineer CODE Summit — The conference mentioned in the opening. Timestamps 0:00 One million AI Engineer subscribers and the CODE Summit 0:51 Mihail Eric and Stanford CS146S 5:26 What junior engineers need to learn now 10:02 Open-source maintainers review the students’ code 15:04 A tour of the ten-week syllabus 24:04 The loop, tools and harness inside a coding agent 32:18 Context engineering for a large codebase 38:15 Skills that make you understand the diff 46:42 The Monaco.com dispute and the airplane banner 50:23 What Monaco does for sales teams 59:21 Inside Monaco’s engineering team 1:01:40 “You buy to learn, and you build to keep” 1:05:24 Making repos agent-ready and closing the on-call loop 1:12:37 When code review becomes the bottleneck 1:20:37 Security, dependencies and permissions 1:30:13 Background agents and Stripe’s Minions 1:34:16 MCP portals and why infrastructure converges 1:41:22 Building the software factory 1:45:24 What work is left for developers? 1:52:16 From personal adoption to team autonomy Opening excerpt from The Thinking Mode, EO Studio .

Latent Space · 9 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
73–78 / 106
Trang 13 / 18