AllyshipContent Radar

Hands-on 278

Công cụ, thử nghiệm và hướng dẫn thực hành

Crawl live-1795090500000 · 19:15 19 thg 11, 2026 · Xuất bản 19:35 19 thg 11, 2026

31

they need to work more on the TUI

56.8

Hướng dẫn luongnv89

they need to work more on the TUI > there are so many places that you need to type to set a ENUM value (take edit mode for example: auto, smart, ... you need to type :| ), some even with full json object which is impossible to view in small TUI dialog > the theme selection, you only can see the change after pressing enter, the theme should be updated while I am moving around. > the setup sometime quite confuse, here I have no idea on how to select mistral-large-4 btw, the mistral-large-4 seems to be OUT at the momement, it fall back to mistral medium 3.5 which I need to re-work with my Claude :(

X · 18 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
33

Cut generative AI costs: Code for rules, models for judgment

56.7

Hướng dẫn Meghan Kothari

<p>The biggest cost lever in a generative AI system is whether you call a model at all. We often see teams route every task through a large language model (LLM), including input validation, formatting, and routing. Each call is cheap in isolation, but at production scale these small costs become one of the largest line items in the architecture. Wherever a deterministic rule can do the job, skipping the model saves money, cuts latency, and removes a source of non-determinism. In this post, you learn how to decide whether each step in a workflow needs a model at all. You can then map each option to AWS: deterministic code, a constrained decision call, or a right-sized model on <a href="https://aws.amazon.com/bedrock/" target="_blank" rel="noopener">Amazon Bedrock</a>.</p> <h2 id="why-unneeded-model-calls-add-up">Why unneeded model calls add up</h2> <p>The pattern usually starts in a prototype. A model call works for one step, so it becomes the default for the next. Those steps include validating input, reformatting a request body, and picking a route. At prototype volume, nobody notices the cost.</p> <p>At production volume, unneeded model calls cost you in three ways:</p> <ul> <li>Spend grows with request count, even for low-value steps.</li> <li>Latency stacks, because every model call in a chain adds hundreds of milliseconds.</li> <li>Output can vary, because the same input can return a different answer. That makes testing harder and audits slower.</li> </ul> <p>Each new model launch then restarts the debate about which model to use. The earlier question, whether the step needs a model at all, goes unasked.</p> <p>Cost guidance, including the <a href="https://docs.aws.amazon.com/wellarchitected/latest/generative-ai-lens/genperf02-bp03.html" target="_blank" rel="noopener">AWS Well-Architected Generative AI Lens</a>, often starts at model selection. This post starts one step earlier: Does the step need a model at all?</p> <h2 id="how-to-decide-whether-a-step-needs-a-model">How to decide whether a step needs a model</h2> <p>Figure 1 shows the whole decision as a flowchart with three labeled lanes: Runs as code, Questions, and Runs on a model. Take each workflow step through three questions in order, and stop at the first option that meets the step’s requirements:</p> <ol type="1"> <li>Can a rule check success, such as a format, a schema, a keyword, or a range? If a rule settles only part of the step, a fast path runs the rule first.</li> <li>Can a model write the rule once, such as a regular expression or a JSON schema?</li> <li>Does the step pick one answer from a fixed list, such as a route, a class, or yes or no?</li> </ol> <p>If the answer to all three is no, the step is open-ended and runs on a right-sized model.</p> <div style="width: 810px" class="wp-caption alignnone"> <a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/17/ML-22259-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/17/ML-22259-1.png" alt="Flowchart that sorts a workflow step into code, a constrained decision call, or a right-sized Amazon Bedrock model" width="800"></a> <p class="wp-caption-text">Figure 1: Deciding whether a workflow step needs a model, and which one</p> </div> <p>Consider a customer support assistant. The following table shows how five of its steps go through the chart.</p> <table border="1px" width="100%" cellpadding="10px"> <tbody> <tr> <td><strong>#</strong></td> <td><strong>Step</strong></td> <td><strong>Path through the chart</strong></td> <td><strong>Where it runs</strong></td> </tr> <tr> <td>1</td> <td>Check an order number’s format</td> <td>A rule can check success.</td> <td><a href="https://aws.amazon.com/api-gateway/" target="_blank" rel="noopener">Amazon API Gateway</a> REST API request validation, or an <a href="https://aws.amazon.com/lambda/" target="_blank" rel="noopener">AWS Lambda</a> function</td> </tr> <tr> <td>2</td> <td>Route a message by intent</td> <td>A rule settles part of it. A keyword match on the fast path resolves messages such as “refund”, and the residual goes to the third question.</td> <td>An <a href="https://aws.amazon.com/step-functions/" target="_blank" rel="noopener">AWS Step Functions</a> Choice state that uses JSONata. The residual goes to a constrained decision call.</td> </tr> <tr> <td>3</td> <td>Extract a tracking number from a carrier email</td> <td>A model can write the rule once. It drafts a regular expression, and you review and commit it.</td> <td>Code, as a committed pattern in AWS Lambda</td> </tr> <tr> <td>4</td> <td>Pick one of four support queues for a ticket</td> <td>The answer comes from a fixed list.</td> <td>A constrained decision call. Unsure answers escalate to the right-sized model.</td> </tr> <tr> <td>5</td> <td>Draft a reply to the customer</td> <td>The task is open-ended.</td> <td>The right-sized model, the smallest Amazon Bedrock model that meets the requirements</td> </tr> </tbody> </table> <p>You can use a model when a task is open-ended, ambiguous, or generative: summarizing a document, drafting a reply, extracting meaning from prose. Use deterministic code when the task has a checkable rule: an email format, a schema, an exact keyword, a numeric range. The following table shows what you trade away when you send a rule-shaped task to a model.</p> <table border="1px" width="100%" cellpadding="10px"> <tbody> <tr> <td><strong>Property</strong></td> <td><strong>Deterministic code</strong></td> <td><strong>Model call</strong></td> </tr> <tr> <td>Cost per invocation</td> <td>Typically orders of magnitude lower</td> <td>Per-token, non-trivial at scale</td> </tr> <tr> <td>Latency</td> <td>Single-digit milliseconds or less</td> <td>Hundreds of milliseconds</td> </tr> <tr> <td>Correctness on a well-specified rule</td> <td>Exact for the rule as written</td> <td>High, but with a non-zero error rate</td> </tr> <tr> <td>A

AWS Machine Learning Blog · 18 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
34

Amazon SageMaker AI real-time endpoints now support KV cache aware routing

56.7

Hướng dẫn Dmitry Soldatkin

<p>Today, Amazon SageMaker AI introduces KV cache-aware routing (KV-aware routing), a new routing strategy for real-time endpoints. It uses the cache state that the model container reports. It sends each request to an instance that already holds a long cached match for its prompt and isn’t too busy. It’s for teams that serve large language models (LLMs) with multi-turn conversations, long shared prompts, or heavy load. You turn it on in two places: add one environment variable to your model so the container reports its cache contents, and set <code>RoutingStrategy</code> to <code>KV_AWARE</code> in the endpoint configuration. Your clients call the endpoint with the same APIs as before.</p> <p>Here’s the problem it solves: LLM applications such as chat assistants, Retrieval Augmented Generation (RAG), and agents with long system prompts send many requests that start with the same text. A model keeps the work it did on a prompt in GPU memory, the KV cache, and skips that work when the same start reaches the same instance again. In <a href="https://aws.amazon.com/blogs/machine-learning/reduce-llm-latency-with-prefix-aware-routing-on-amazon-sagemaker-inference/" target="_blank" rel="noopener">part 1 of this series</a>, we introduced prefix-aware routing for Amazon SageMaker AI, which sends requests that start the same way to the same instance. Prefix-aware routing infers where a prefix is cached from the request alone, so it can’t tell when an instance has evicted that cache.</p> <p>Prefix-aware routing picks an instance from the request alone. It hashes the start of the request, up to the <code>PrefixLength</code> you configure (bytes of the <code>InvokeEndpoint</code> payload, or characters of the message text for the OpenAI-compatible Chat Completions API), and sends requests that start the same way to the same instance. It never sees what an instance actually has cached. This works well when every request starts with the same shared text, and less well in two cases:</p> <p>The cache on an instance changes over time. An instance evicts older KV cache entries when its GPU memory fills up, but the router keeps sending matching requests there as if the cache were still warm.</p> <p>A conversation starts out short. In a multi-turn chat, each turn adds to the history. While the request is still shorter than <code>PrefixLength</code>, every new turn changes the text the router hashes, so the next turn can land on a different instance than the one that cached the earlier turns. The turns of a conversation stay on one instance only after the request grows past <code>PrefixLength</code>.</p> <p>In this post, we explain how KV-aware routing works, when to use it, and how to enable, invoke, and verify it on single model endpoints and inference component endpoints.</p> <h2 id="how-kv-aware-routing-works">How KV-aware routing works</h2> <p>KV-aware routing has three parts: the model container reports what it has cached, SageMaker AI keeps a managed cache index of that state for the endpoint, and the endpoint uses the index to choose an instance for each request.</p> <p>When you turn events on with <code>SM_VLLM_KV_EVENTS_CONFIG</code>, the container publishes a KV cache event each time it stores or evicts cache entries, and SageMaker AI uses these events to keep the index current. The index holds only pointers: one-way, truncated hashes of cached content, each mapped to the instance or inference component copy that holds it. It never stores prompts, token IDs, request payloads, or KV tensors. When a request arrives, the endpoint tokenizes the prompt and looks up the tokens in the index. The index returns the instances or inference component copies that hold part of the prompt in their cache, and how much of it each one holds. For example, if instance A holds the cached start of a conversation, the next turn of that conversation goes to A. Because the index matches on tokens rather than a fixed number of bytes, a multi-turn history can match in full as it grows. A request that shares only part of its prompt still goes to the instance with the longest match.</p> <p>The endpoint then picks among these candidates by weighing how much of the prompt each one has cached against how busy it is. A candidate that is too busy is skipped, so if the instance with the longest match is busy, the request goes to the next-best one that is available. If no candidate has a useful match or all of them are busy, the request falls back to load-aware routing, as described in Routing overhead and fallbacks. You don’t configure a concurrency threshold for this, as you do with prefix-aware routing. The index also follows the fleet. When you scale in, entries for removed instances are dropped. When you scale out or an update replaces instances, the new instances start with empty caches, so latency can be higher for a short time.</p> <div style="width: 810px" class="wp-caption alignnone"> <a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/17/ML-22381-1-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/17/ML-22381-1-1.png" alt="KV-aware routing: the router uses the managed cache index to send each request to the instance holding the longest cached match" width="800"></a> <p class="wp-caption-text">Figure 1: KV-aware routing on a SageMaker AI endpoint. The router looks up the managed cache index and forwards the request to the instance that holds the longest cached match. Each instance reports KV cache events back to the index</p> </div> <p>In Figure 1, the endpoint has three instances, A, B, and C. Each instance runs the same model in its own model container (the AWS vLLM Deep Learning Container listed in Requirements and limits) and has its own KV cache in GPU memory, so each instance holds different cached content: A holds turns 1 to 3 of conversation 1, B holds conversations 2 and 3, and C holds doc

AWS Machine Learning Blog · 18 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
35

Putting a spotlight on AI, society, and humanity

56.7

Hướng dẫn Danna Lorch | School of Humanities, Arts, and Social Sciences

<p dir="ltr">MIT students are going to office hours less often. They’re spending less time working through problem sets, and with one another. Those were among the findings of MIT’s recent report on&nbsp;<a href="https://aiandeducation.mit.edu/report/">artificial intelligence and education</a>, and they are one reason a growing number of MIT faculty are concerned about how generative AI is both changing how people work and affecting how they think, learn, and relate to one another.</p><p dir="ltr">For the&nbsp;<a href="https://mithic.mit.edu">MIT Human Insight Collaborative</a> (MITHIC), those questions have become a central focus, and MITHIC is working with collaborators across the Institute, including the<a href="https://computing.mit.edu/cross-cutting/social-and-ethical-responsibilities-of-computing/"> Social and Ethical Responsibilities of Computing</a> (SERC), to help MIT address them.</p><p dir="ltr">Launched in 2024 as a strategic initiative, MITHIC brings together scholars, artists, educators, and researchers to explore the human dimensions of the defining challenges of our time. Since its launch, MITHIC has supported <a href="https://mithic.mit.edu/impact/">67 projects across 31 MIT units</a>, offering a total of $5.5 million in funding.</p><p dir="ltr">Roughly a third of those projects are focused on AI, society, and humanity — a topic that MITHIC faculty co-lead <a href="https://anthropology.mit.edu/people/faculty/heather-paxson">Heather Paxson</a> says has become an increasingly important part of the initiative.</p><p dir="ltr">“When we refer to AI and society, we’re not only speaking about individual ethical decision-making or how to retrain workers. We’re looking at how AI is reshaping sociality, how we&nbsp;<a href="https://news.mit.edu/2026/when-we-talk-to-machines-sherry-turkle-book-0929">relate</a> to one another,” says Paxson, a professor of anthropology and the associate dean of the School of Humanities, Arts, and Social Sciences (SHASS).&nbsp;</p><p dir="ltr"><strong>Eroding “creative friction”&nbsp;</strong></p><p dir="ltr">Building upon this focus, an&nbsp;<a href="https://news.mit.edu/2026/exploring-societal-impacts-of-ai-0623">AI and Society Forum</a> took place on campus earlier this year. Co-organized by MITHIC, SHASS, SERC, and the<a href="https://genai.mit.edu/"> MIT Generative AI Impact Consortium</a> (MGAIC), the event featured presentations and panels concentrating on jobs, democracy, and AI.&nbsp;</p><p dir="ltr">On campus, the report on AI and education warned that AI is eroding the “creative friction” that has long defined an MIT education: students debating ideas and challenging one another, their professors, and their teaching assistants, in class and out of it.</p><p dir="ltr">MITHIC faculty co-lead <a href="https://cmsw.mit.edu/profile/seth-mnookin/">Seth Mnookin</a>, a science writing professor and head of Comparative Media Studies Program/Writing (CMS/W), said, “The report wasn’t saying that AI is here and we all need to adapt. It was saying that we need to be asking more fundamental questions.”&nbsp;&nbsp;</p><p dir="ltr">“When the United States entered World War II in 1941, the Institute’s newly minted scientists and engineers helped shape that era through science and technology,” Mnookin said. “We’re at a similarly consequential moment with AI. The difference is that this time, the questions that matter most can’t be answered by technologists alone. That’s where the humanities, arts, and social sciences come in.”</p><p><strong>Crossing disciplines in the pursuit of answers</strong></p><p>MITHIC is supporting a range of interdisciplinary projects through its <a href="https://mithic.mit.edu/shass-connectivity-fund/" target="_blank" rel="noreferrer noopener">SHASS+ Connectivity Fund</a>, in which a SHASS scholar partners with a researcher in another part of MIT.&nbsp;</p><p>In the context of the 2026 U.S. midterm elections, political science professors <a href="https://polisci.mit.edu/people/adam-berinsky" target="_blank" rel="noreferrer noopener">Adam Berinsky</a> and <a href="https://polisci.mit.edu/people/charles-stewart-iii" target="_blank" rel="noreferrer noopener">Charles Stewart III</a> and MIT Sloan School of Management Assistant Professor <a href="https://mitsloan.mit.edu/faculty/directory/chara-podimata" target="_blank" data-type="link" data-id="https://mitsloan.mit.edu/faculty/directory/chara-podimata" rel="noreferrer noopener">Chara Podimata</a> are studying how large language models (LLMs) have become intermediaries of democratic knowledge.&nbsp;</p><p>The ongoing project, which has demonstrated how voters in Alaska’s midterms who conducted their pre-election research using Anthropic’s AI model, Claude, may have received incomplete information about candidates and their views, was recently <a href="https://www.nytimes.com/2026/09/10/business/media/ai-chatbots-election-misinformation.html?unlocked_article_code=1.IVE.8Kr5.eEpKyyfrOLzA&amp;smid=url-share">featured in <em>The New York Times</em></a>.&nbsp;</p><p>“AI is becoming part of the way people encounter and make sense of political information, and yet we know relatively little about what that information environment actually looks like, how it differs for different user demographics, political identities, and geographies,” Podimata told the <em>Times</em>.</p><p>In another project, CMS/W Professor <a href="https://cmsw.mit.edu/profile/eric-klopfer/" target="_blank" rel="noreferrer noopener">Eric Klopfer</a><strong> </strong>and brain and cognitive sciences Professor <a href="https://bcs.mit.edu/directory/pawan-sinha" target="_blank" data-type="link" data-id="https://bcs.mit.edu/directory/pawan-sinha" rel="noreferrer noopener">Pawan Sinha</a> are co-leading research that’s measuring AI’s near- and long-term effects on cognition.</p><p>Meeting at the intersection of AI, education, and neuroscience, the professors are examining whether generative AI is increasing the human brain’s con

MIT News: AI · 18 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
36

If you're worried about AI's effect on your problem solving and coding skills, or how to learn with AI - then look at SolveIt.

56.6

Hướng dẫn chris_thomas_uk

If you're worried about AI's effect on your problem solving and coding skills, or how to learn with AI - then look at SolveIt. Concise by design, you can edit the AI's responses, allows close reading, and build skill mastery with AI. Thank you @jeremyphoward and @answerdotai

X · 18 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
31–36 / 179
Trang 6 / 30