AllyshipContent Radar

News 2802

Chủ đề cộng đồng đang thảo luận

Crawl live-1795083300000 · 17:15 19 thg 11, 2026 · Xuất bản 17:31 19 thg 11, 2026

156 / 2802 chủ đề đạt ngưỡng 55 trong khoảng này.

142

The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements

70

11 tiếng nói · chưa gắn nhãn trưởng lab · 1 nền tảng · 14 tương tác đã đo · Nguồn crawl

Yuanhao Zeng, Shaocheng Lan và 9 nguồn khác 1 bài viết · Hugging Face Papers
Biên tập tiếng Việt
Chi tiết nguồn

Bài báo mô tả vòng lặp tiến hóa tác nhân: proposer tạo các ứng viên (prompts, skills, memories) và selection chọn những ứng viên có điểm cao hơn. Tuy vậy, họ nói rằng evolution có thể thất bại khi khởi động trên một số tác vụ vì đánh giá nhiễu không thể phân biệt đáng tin cậy giữa ứng viên tốt và xấu.

Agent evolution follows a simple loop: a proposer writes candidate artifacts (prompts, skills, memories) and selection accepts those that score higher. This loop has shown promise, yet practitioners face a puzzle: methods trade wins across papers, and evolution often fails to bootstrap on certain tasks. We trace this puzzle to the Resolution Hypothesis: agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them. Across seven evolution methods and six models (over 10M LLM calls), we find support for the hypothesis at both levels where evaluation scores are used. At the candidate level (choosing artifacts), helpful candidate artifacts exist, but selection is no better than random at telling them from harmful ones, so errors accumulate across rounds. At the method level (ranking methods), typical margins of a few percentage points are smaller than the difference between two independent runs of the same method, so published rankings often rest on luck. Together, these findings indicate that agent evolution is often bottlenecked by reliably identifying improvements. This calls for a shift in how we develop and evaluate self-improving agents, from designing ever more candidate improvement systems to making true improvements detectable and accumulable.

  • We trace this puzzle to the Resolution Hypothesis: agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them.
  • We trace this puzzle to the Resolution Hypothesis: agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them.
  • Across seven evolution methods and six models (over 10M LLM calls), we find support for the hypothesis at both levels where evaluation scores are used.
  • Together, these findings indicate that agent evolution is often bottlenecked by reliably identifying improvements. This calls for a shift in how we develop and evaluate self-improving agents, from designing ever more candidate improvement systems to making true improvements detectable and accumulable.
143

Decoding Without a Decoder: Direct Pixel Prediction from Diffusion Transformers

69

7 tiếng nói · chưa gắn nhãn trưởng lab · 1 nền tảng · 9 tương tác đã đo · Nguồn crawl

Hansheng Chen, Shengqu Cai và 5 nguồn khác 1 bài viết · Hugging Face Papers
Nguyên văn nguồn
Chi tiết nguồn

Latent diffusion models rely on a decoder to map denoised latents to pixels, adding complexity and inference cost. We introduce Decoding Without a Decoder (DWD), which lets a finetuned diffusion transformer produce final pixels directly through a linear head. Unlike previous methods, where the denoising output leads to final pixels, DWD separates the two roles: the original denoising head continues to drive sampling, then the new DWD head predicts the pixels at the final step. We extend pretrained latent models with DWD efficiently by finetuning only the last few transformer blocks, training the DWD head with pixel quality losses while anchoring the denoising head to the pretrained model to preserve the sample trajectory. Across latent image and video generators, DWD removes the decoder and reduces inference cost while maintaining or even improving pixel quality. On Wan video generation, it visibly reduces detail distortion and flickering, lowering warping error by 23%. Remarkably, the same separation also improves pixel diffusion models that do not even use a decoder: a final-step DWD head produces substantially better texture detail in text-to-image generation, and improves ImageNet FID from 1.55 to 1.43, a new state of the art among pixel flow models.

144

QDA: Closed-Loop LLM Quantization Design with Agents

69

4 tiếng nói · chưa gắn nhãn trưởng lab · 1 nền tảng · 9 tương tác đã đo · Nguồn crawl

Xingyang Li, Tianlang Zhao và 2 nguồn khác 1 bài viết · Hugging Face Papers
Biên tập tiếng Việt
Chi tiết nguồn

Bài báo nêu rằng post-training quantization là cách tiêu chuẩn để phục vụ LLM với chi phí thấp, nhưng công thức quantization tốt phụ thuộc vào từng mô hình và độ chính xác, nên chuyên gia phải thiết kế lại cho mỗi mô hình. Họ giới thiệu QDA, nơi một agent thiết kế recipe cho từng mô hình và độ chính xác. QDA đánh giá nhanh bằng cách chấm candidate theo KL divergence đầu ra so với mô hình gốc trên reasoning traces mà agent không thấy, và kiểm tra thêm các candidate hứa hẹn bằng long generations để bắt lỗi như repetition loops.

Post-training quantization is the standard way to serve large language models at low cost. However, the best quantization recipe differs across models and precisions. As a result, experts must redesign it for every new model, which is time-consuming and often sub-optimal. Fortunately, an LLM agent can take over this design by refining the recipe over rounds of building and evaluating candidates, so that more rounds lead to better recipes. However, evaluating one quantized reasoning model takes tens of GPU-hours, and combining methods from different codebases means porting each to the new architecture by hand. We present QDA, a system in which an agent designs the recipe for each model and precision. To make evaluation fast, QDA scores each candidate in minutes by its output KL divergence from the original model on reasoning traces that the agent never sees, and checks promising ones with long generations to catch failures such as repetition loops. To make building fast, QDA provides a library of public PTQ methods that are compatible with one another and with new model architectures. Across four reasoning models, QDA improves accuracy over the strongest public recipe by 0.9, 1.4 and 2.2 percentage points on average at W4A16, W4A8 and W3A16, using 64 GPU-hours per search. On the 320B-parameter MoE model GLM-5.3-Flash at NVFP4 W4A4, it recovers 98.3% of the FP8 model's accuracy, against 96.9% for the best vendor checkpoint. Every QDA checkpoint also runs on existing serving kernels with no inference overhead. We will release the library, the evaluator and the harness.

  • Across four reasoning models, QDA improves accuracy over the strongest public recipe by 0.9, 1.4 and 2.2 percentage points on average at W4A16, W4A8 and W3A16, using 64 GPU-hours per search.
  • Across four reasoning models, QDA improves accuracy over the strongest public recipe by 0.9, 1.4 and 2.2 percentage points on average at W4A16, W4A8 and W3A16, using 64 GPU-hours per search.
  • On the 320B-parameter MoE model GLM-5.3-Flash at NVFP4 W4A4, it recovers 98.3% of the FP8 model's accuracy, against 96.9% for the best vendor checkpoint.
  • Every QDA checkpoint also runs on existing serving kernels with no inference overhead.
145

Let Samples Sing in Chorus: Mitigating Spurious Correlations via Cross-view Consensus

66

6 tiếng nói · chưa gắn nhãn trưởng lab · 1 nền tảng · 3 tương tác đã đo · Nguồn crawl

Weiwei Li, Junzhuo Liu và 4 nguồn khác 1 bài viết · Hugging Face Papers
Nguyên văn nguồn
Chi tiết nguồn

Deep models often rely on spurious correlations that fail on underrepresented groups. Existing methods either correct human-specified bias attributes or infer unknown biases from the learned representation space. However, annotated attributes may differ from the shortcuts actually used by a model, while a single representation provides incomplete and potentially noisy evidence. Our key insight is that a bias obscured in one feature space may be clearer in another. In this work, we propose CHORD (Cross-view Harmonic Outlier Reversal for Debiasing), a multi-view framework that allows samples to "sing in chorus" across diverse representation spaces to achieve robust cross-view consensus. CHORD aggregates aggregates within-class partitions in each view, which is a proxy for specifying candidate biases, to debias the target representation and classifier without bias supervision. Experiments on seven benchmarks across vision, language, speech, and medical imaging show consistent improvements in worst-group accuracy. Code and checkpoints are available at https://github.com/davelee-uestc/chord_debiasing_public .

146

Out of Sight, Still in Mind: Token Compression for Omni-LLMs

65

4 tiếng nói · chưa gắn nhãn trưởng lab · 1 nền tảng · 2 tương tác đã đo · Nguồn crawl

Suho Yoo, Youngjoon Jang và 2 nguồn khác 1 bài viết · Hugging Face Papers
Nguyên văn nguồn
Chi tiết nguồn

The goal of this paper is to reduce the input token cost of Omni-modal large language models (Omni-LLMs) at inference time. Omni-LLMs reason jointly over audio, video and text, but the cost of the three streams is highly unbalanced: visual tokens account for the vast majority of the input, and are highly redundant. In this paper, we propose ReMo, a training-free framework that compresses visual tokens by redistributing their information across modalities: a visual token is kept only if its information appears nowhere else. ReMo achieves this in two ways: (i) it aligns audio and video in a shared space via closed-form calibration, and removes visual tokens already explained by the audio or by other visual tokens; and (ii) it replaces object-level visual tokens, identified by a lightweight object detector, with compact text proxies, short descriptions of each object and its location, conveying the same content in far fewer tokens. On Qwen2.5-Omni at two model scales, ReMo removes 54% of the input tokens with comparable accuracy. It performs comparably to the full-token model, reaching 101.2% and 101.4% of its average accuracy over five audio-visual benchmarks.

147

NexSight: Verifiable Tasks for Fine-Grained Agentic Long-Video Understanding

65

4 tiếng nói · chưa gắn nhãn trưởng lab · 1 nền tảng · 2 tương tác đã đo · Nguồn crawl

Tianci Xie, Li Chen và 2 nguồn khác 1 bài viết · Hugging Face Papers
Biên tập tiếng Việt
Chi tiết nguồn

Bài báo nói rằng các câu hỏi chi tiết theo thời gian trong video dài có thể yêu cầu tìm các khoảnh khắc ngắn rải rác trong nhiều giờ. Việc đánh giá và học cho các mô hình tác nhân quyết định khi nào và nơi nào cần quan sát là khó vì bằng chứng thưa thớt có thể không đảm bảo đếm đúng hoặc không đủ để kiểm chứng toàn bộ dòng thời gian. Nhóm giới thiệu NexSight để xây dựng các tác vụ có thể kiểm chứng nhằm vừa đánh giá vừa huấn luyện hiểu video dài theo dạng tác nhân.

Fine-grained questions about long videos ask how many times an action occurs, in what order events unfold, or exactly when a state changes. Answering them can require finding brief moments scattered across hours of footage. Agentic models that decide when and where to look promise to find these moments, yet this ability is hard both to measure and to learn. Sparse supporting evidence may establish that an action occurred without establishing its total count; verifying timeline-wide claims requires checking for missed occurrences and double counting. Answer-only supervision, in turn, does not explicitly supervise the search for evidence. We present NexSight, which builds verifiable tasks to both evaluate and train agentic long-video understanding. Its agentic pipeline, SightForge, accepts a task only if a verifier blind to the draft answer can reconstruct it from evidence densely reacquired from the raw video and counterexample checks fail to refute it. This yields SightLine, 310 evidence-traceable tasks from 108 long videos, evaluated through SightLoop, a shared multiscale harness supporting temporal exploration, local inspection, evidence integration, and revisiting while recording returned observations. On disjoint videos, solver trajectories on verified tasks become step-level supervision for observation decisions. The strongest model reaches only 54.52% on SightLine, yet 79.35% when given the annotated evidence. With under 6K examples, Sight-35B nearly doubles its base model's interactive accuracy (21.29% to 42.26%), outperforms the 397B model of its family, begins using explicit spatial inspection after post-training, which its base model never uses, and gains over 8 points on two external long-video benchmarks.

  • This yields SightLine, 310 evidence-traceable tasks from 108 long videos, evaluated through SightLoop, a shared multiscale harness supporting temporal exploration, local inspection, evidence integration, and revisiting while recording returned observations.
  • This yields SightLine, 310 evidence-traceable tasks from 108 long videos, evaluated through SightLoop, a shared multiscale harness supporting temporal exploration, local inspection, evidence integration, and revisiting while recording returned observations.
  • The strongest model reaches only 54.52% on SightLine, yet 79.35% when given the annotated evidence.
  • With under 6K examples, Sight-35B nearly doubles its base model's interactive accuracy (21.29% to 42.26%), outperforms the 397B model of its family, begins using explicit spatial inspection after post-training, which its base model never uses, and gains over 8 points on two external long-video benchmarks.
148

Thinking Harder Doesn't Lead to Better Choices: Reasoning Effort Increases the Confidence but Not the Correctness of LLM Shopping Agents

57

2 tiếng nói · chưa gắn nhãn trưởng lab · 1 nền tảng · 2 tương tác đã đo · Nguồn crawl

Adi Krishnamoorthy, X. Y. Han 1 bài viết · Hugging Face Papers
Nguyên văn nguồn
Chi tiết nguồn

Large language models (LLMs) are increasingly deployed as autonomous shopping agents that process multimodal product listings (consisting of visual and text data) and recommend purchases on behalf of consumers. Naturally, one might assume that models with more thinking power would make higher-quality purchasing decisions. To test this belief, we ask two questions about such agents: does asking a model to think harder make it choose better products, and does the confidence it reports tell us the likelihood it may be wrong? We benchmark ten frontier models on three product-choice benchmarks built from WebShop, Amazon's SQID and Wayfair's WANDS platforms to see if LLMs with more thinking power actually make better-quality product purchase decisions. Across all three benchmarks, we found that increasing the thinking effort does not increase decision quality in a statistically meaningful way. The same change in thinking effort raises stated confidence at a statistically significant level, so extra thinking buys confidence rather than correctness. We also measure the calibration of these models (how well their reported confidence percentage matches their true correctness percentage) in these tasks and show that most models are not well-calibrated at every effort level. We also find that errors are strongly shared: when one model errs in a task, several others tend to err on that same task. Furthermore, on tasks where several models all err, a large number of them pick the same wrong product.

149

When One GPU Isn’t Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM

61

2 tiếng nói · chưa gắn nhãn trưởng lab · 2 nền tảng · chưa có số tương tác · Nguồn crawl

Fatima Fall, vLLM 1 bài viết · NVIDIA Developer Blog, X
Nguyên văn nguồn
Chi tiết nguồn

Running a frontier-scale model locally means keeping all its weights in GPU memory; the largest models no longer fit on one machine. Two NVIDIA DGX Stations... Running a frontier-scale model locally means keeping all its weights in GPU memory; the largest models no longer fit on one machine. Two NVIDIA DGX Stations connected over ConnectX-8 RDMA serve one model through a single endpoint, with tensor parallelism splitting the model weights and computation across both GPUs. The NVIDIA GLM-5.3 NVFP4 model, a quantized version of Z.ai’s mixture-of… Source

150

Show HN: One cloud computer for all your agents

68

2 tiếng nói · chưa gắn nhãn trưởng lab · 2 nền tảng · 11 tương tác đã đo · Nguồn crawl

taro666, omarsar0 1 bài viết · HN: Claude, X
Nguyên văn nguồn
Chi tiết nguồn

Hi HN, my team just released box by Simular, a cloud computer for all your agents, with fast computer use built-in. box gives your agents one remote Ubuntu desktop with Chrome and a shell, which your AI can then drive over MCP while your own computer stays unoccupied. That means you can turn your basic ChatGPT plan into a personal agent by giving it a box computer; let your Claude Code pick up what Codex has left off on the same box to maximize usage; actually close your laptop and let tasks run in one persistent, 24/7 environment with all your logins and setup. Some box demos: https://x.com/SimularAI/status/2121329778896871665?a=U4U1yYT... We started on this because we believe computers shouldn't need human operation in the future. When agents arrived, people started running them on mini computers and you see colleagues walk around the office with their laptop lid open. We think there's a better solution - cloud computers engineered for agents. Simular started three years ago to build out the full stack of an autonomous computer. box is our attempt to give agents a body to live in. Our computer use outperformed Opus and GPT on OSWorld, costing fewer tokens. https://thenewstack.io/sai-agent-osworld-benchmark/ Copy/paste one command into your agent to start: "Set up box from https://box.simular.ai/install " You can watch box's live action from a dashboard https://box.simular.ai/box We'd love for you to try it and share your feedback. I'll be in comments here to answer any question.

151

Hugging Face and OpenRouter join forces to make open-weight models easy to run

65

2 tiếng nói · chưa gắn nhãn trưởng lab · 3 nền tảng · 4 tương tác đã đo · Nguồn crawl

acossta, Hugging Face 1 bài viết · HN: open weights, Hugging Face Blog, X
Nguyên văn nguồn
Chi tiết nguồn
152

Show HN: Fakemetrics.com – "build in public" like a superstar

57

1 tiếng nói · chưa gắn nhãn trưởng lab · 1 nền tảng · 73 tương tác đã đo · Nguồn crawl

tigranbs 1 bài viết · HN: Claude
Nguyên văn nguồn
Chi tiết nguồn

Hi HN, I am Tigran (github.com/tigranbs) I made Fakemetrics.com as a fun experiment while testing how well Claude Opus can build pixel-identical UIs. It turned out it wasn't as easy as just prompting it, but I spent time building the system so it works almost every time. And I am pretty sure you are seeing a bunch of people posting weekend vibe-coded apps and saying "$50K/month after quitting my McDonnalds job...". Well, even my friend tried to fake it a few months back. So, I kinda did this fakemetrics.com as an experiment to see how well I can have the system that clones UIs, and packaged that as a very simple (I think) usable product for the people that want to be a superstar "build in public" type of a person with no revenue, but a perfect Stripe user growth screenshot. Useful? - Probably not that much Fun? - YES!! The best part is that Claude, ChatGPT, or Grok models aren't that good at UI clones by default, but with fakemetrics.com I kinda made a system that produces near-identical results from screenshots fairly fast. That was worth learning! Let me know what you think? Should I move on with the UI cloning as an open-source product?

153

ALIVE: Making Streaming Video Models Interactive

65

16 tiếng nói · chưa gắn nhãn trưởng lab · 1 nền tảng · 12 tương tác đã đo · Nguồn crawl

Jinzhao Li, Yuan Xu và 14 nguồn khác 1 bài viết · Hugging Face Papers
Nguyên văn nguồn
Chi tiết nguồn

Streaming video LLMs can continuously observe the world, yet often handle user requests in isolation. We introduce ALIVE, an interactive streaming video model supporting proactive monitoring, streaming task coordination, and feedback-seeking interaction. We construct ALIVE-SFT-288K and ALIVE-RL-3K for two-stage training. Contrastive supervised fine-tuning teaches when and how to respond, while reinforcement learning with Behavior-Aligned Credit Assignment (BACA) combines trajectory-level and local advantages to refine decisions, responses, and silence across multi-turn interactions. We also introduce FSI-Bench to evaluate feedback seeking and subsequent task completion. ALIVE achieves the highest overall scores among the compared models on four public benchmarks and FSI-Bench, demonstrating strong streaming understanding and interactive capabilities.

154

VAE Rhythm: Temporal Decoding Traces for AI-Generated Video Detection

61

4 tiếng nói · chưa gắn nhãn trưởng lab · 1 nền tảng · 2 tương tác đã đo · Nguồn crawl

Ke Sun, Shuwei Li và 2 nguồn khác 1 bài viết · Hugging Face Papers
Nguyên văn nguồn
Chi tiết nguồn

Visible artifacts are an unreliable cue for detecting generated videos, because newer generators learn to remove them. We instead detect a trace left by the generation pipeline itself. Most video generators decode latents with a 3D causal VAE that groups consecutive frames in time, so the fidelity of generated frames rises and falls periodically with their position in a group. We call this periodic trace VAE Rhythm. Real videos are captured frame by frame and do not have it. To reveal it without training, we propose a phase scan. The scan reconstructs a clip with an off-the-shelf VAE from successive start offsets and measures the periodicity of the reconstruction error across these offsets. We find VAE Rhythm in most of the 39 variants of 31 generators we test, including the closed-source Sora, Kling 3.0 and Seedance 2.0. We use the rhythm as a zero-shot score, as a plug-in for existing detectors and as a data engine that pairs real videos with their VAE reconstructions to train our detector CaDet. On DF26, the training-free rhythm score outperforms all 15 retrained baselines, and as a plug-in it improves all 19 tested detectors by 6.7% to 23.7% AUROC. CaDet achieves the best AUROC on AIGVDBench, DF26 and three 2026 closed-source models, and its variant trained without any generated video still exceeds 90% on AIGVDBench.

155

Read the Stand with Jensen letter and consider adding your name. #standwithjensen https://standwithjensen.com/

56

4 tiếng nói · chưa gắn nhãn trưởng lab · 1 nền tảng · chưa có số tương tác · Nguồn crawl

amasad, petergyang và 2 nguồn khác 0 bài viết · X
Nguyên văn nguồn
Chi tiết nguồn

Read the Stand with Jensen letter and consider adding your name. #standwithjensen https://standwithjensen.com/

156

Understanding Positional Embeddings in Modern LLMs

55

2 tiếng nói · chưa gắn nhãn trưởng lab · 2 nền tảng · chưa có số tương tác · Nguồn crawl

Sebastian Raschka, PhD, Sebastian Raschka 1 bài viết · Ahead of AI, X
Nguyên văn nguồn
Chi tiết nguồn

This article explains how LLMs encode and process token position information by covering classic and contemporary methods like RoPE. It’s essentially a comprehensive journey where we will work through the mechanisms with visual examples, a little math, and a small PyTorch implementation. And, of course, we will then also look at how modern architectures adapt or omit explicit positional encodings. Below, you can find a preview of which positional embedding choices are used in 2026 open-weight models. We will discuss these choices later after covering the underlying mechanisms. Figure 1: A preview of the gallery’s 2026 architectures, grouped by positional encoding. Architecture figures taken from my LLM gallery . But why spending so much time writing/reading about this rather minor building block? LLMs keep evolving, and as I was preparing a new article on the latest LLM architecture trends this year, I realized that I never covered the positional information methods used in LLMs in any article so far. So, instead of letting the other article get out of hand, I thought it might make sense to publish this as a standalone building block for the upcoming LLM architecture and trends analysis. Also, over the years, many readers have asked me for a tutorial on relative positional embeddings (aka RoPE) to follow up on the learned absolute embeddings used in GPT. It’s been on my TODO list for a long time, so I finally revisited my old draft and put this walkthrough together, which will cover different positional methods in LLMs; (finally) how RoPE works; what methods are used by recent open LLMs. Note that this article assumes some familiarity with basic LLM architectures and works well as a follow-up to Build a Large Language Model (From Scratch) . If the concepts and computation of the attention internals, particularly query, key, and value vectors, are a little fuzzy, I recommend checking my self-attention walkthrough as a refresher: The article is also very technical and likely too detailed for some. However, I really wanted to have this foundation out there for future articles to reference. And no worries, the next one will return to a more comfortable style. PS: I started experimenting with Substack’s new “perks” system. As a thank-you to supporters, I’ve added image exports and metadata from the architecture gallery for you to use in your own analyses. PPS: If you are already familiar with how position embeddings work, feel free to fast-forward to Section 6, Positional choices in modern architectures . Note that this article may be best read in a web browser than an email client. I recommend checking out the web view here , which also has a navigation bar on the left side. 1. Where position information enters an LLM Let’s start with the overall architecture. A tokenizer encodes the input text into token IDs. Next, the input token embedding layer of an LLM maps each of these IDs to a vector of numbers. We usually refer to these entries in that vector as “features” (similar to classic ML). The transformer blocks inside the LLM then modify these vectors using attention and feedforward layers. Lastly, the model’s output layer produces logit scores (converted into probabilities) for the next token. The following shows this path in GPT-2 (a relatively classic LLM) and MiniMax-M3 (a more recent LLM released this year; a placeholder for most other recent LLMs). As we can see, the architectures are not too different from each other, but before we get to the positional information parts, let’s focus on the orange blocks for now (the token embedding layers) and the transformer blocks above them; the other architecture differences aren’t needed for this explanation. Figure 2: Token embedding layers in GPT-2 and MiniMax-M3, highlighted in orange. (Side note: M3’s first three out of 60 feedforward layers are dense not MoEs.) A token embedding layer, shown in the figure above, maps each token ID to a vector. Position information is then added afterward. Concretely, and as we will discuss in more detail later, GPT-2 adds a position vector to each token embedding before the modified embedding enters the first transformer block. On the other hand, MiniMax-M3 adds the positional information in each transformer block using a method called RoPE inside attention. We will talk about these methods in more detail later, but first, let’s start with the big picture information explaining why explicit positional information is added or needed. 1.1 Why add positional information? Consider the following two input sentences as a simple example: “The red fox jumps over the gray fox.” “The gray fox jumps over the red fox.” Changing the word order changes which fox does the jumping, as illustrated in the following figure. Figure 3: Changing the word order changes which fox jumps. For the examples in this article, we will treat each word as one token for simplicity. (As you know, real tokenizers can split words into smaller subword tokens.) The first sentence then has eight token positions, from 0 through 7. Figure 4: Token IDs and positions in the toy sentence. # Assume the following small vocabulary. # Real vocabularies typically have 50k to 200k entries vocab = { "The": 0, "red": 1, "fox": 2, "jumps": 3, "gray": 4, "over": 5, "the": 6, } (If you are interested in how vocabularies are built, check out my tokenizer materials on GitHub .) Note that “fox”, which appears twice in the input sentence, gets the same token ID 2 in both cases. And since they both have the same token ID, irrespective of the position in the input sentence, they would get the same embedding vector from the embedding layer. (For those who are interested in more details on how this selection from the embedding table looks like, check Appendix A .) Now, let’s consider a regular self-attention mechanism with no positional encoding (and no causal mask), where every input token can attend to every other token in the input. In this scenario, swapping any tw

141–156 / 156
Trang 8 / 8