AllyshipContent Radar

Hands-on 278

Công cụ, thử nghiệm và hướng dẫn thực hành

Crawl live-1795090500000 · 19:15 19 thg 11, 2026 · Xuất bản 19:35 19 thg 11, 2026

55

Understanding Positional Embeddings in Modern LLMs

52.0

Hướng dẫn Sebastian Raschka, PhD

This article explains how LLMs encode and process token position information by covering classic and contemporary methods like RoPE. It’s essentially a comprehensive journey where we will work through the mechanisms with visual examples, a little math, and a small PyTorch implementation. And, of course, we will then also look at how modern architectures adapt or omit explicit positional encodings. Below, you can find a preview of which positional embedding choices are used in 2026 open-weight models. We will discuss these choices later after covering the underlying mechanisms. Figure 1: A preview of the gallery’s 2026 architectures, grouped by positional encoding. Architecture figures taken from my LLM gallery . But why spending so much time writing/reading about this rather minor building block? LLMs keep evolving, and as I was preparing a new article on the latest LLM architecture trends this year, I realized that I never covered the positional information methods used in LLMs in any article so far. So, instead of letting the other article get out of hand, I thought it might make sense to publish this as a standalone building block for the upcoming LLM architecture and trends analysis. Also, over the years, many readers have asked me for a tutorial on relative positional embeddings (aka RoPE) to follow up on the learned absolute embeddings used in GPT. It’s been on my TODO list for a long time, so I finally revisited my old draft and put this walkthrough together, which will cover different positional methods in LLMs; (finally) how RoPE works; what methods are used by recent open LLMs. Note that this article assumes some familiarity with basic LLM architectures and works well as a follow-up to Build a Large Language Model (From Scratch) . If the concepts and computation of the attention internals, particularly query, key, and value vectors, are a little fuzzy, I recommend checking my self-attention walkthrough as a refresher: The article is also very technical and likely too detailed for some. However, I really wanted to have this foundation out there for future articles to reference. And no worries, the next one will return to a more comfortable style. PS: I started experimenting with Substack’s new “perks” system. As a thank-you to supporters, I’ve added image exports and metadata from the architecture gallery for you to use in your own analyses. PPS: If you are already familiar with how position embeddings work, feel free to fast-forward to Section 6, Positional choices in modern architectures . Note that this article may be best read in a web browser than an email client. I recommend checking out the web view here , which also has a navigation bar on the left side. 1. Where position information enters an LLM Let’s start with the overall architecture. A tokenizer encodes the input text into token IDs. Next, the input token embedding layer of an LLM maps each of these IDs to a vector of numbers. We usually refer to these entries in that vector as “features” (similar to classic ML). The transformer blocks inside the LLM then modify these vectors using attention and feedforward layers. Lastly, the model’s output layer produces logit scores (converted into probabilities) for the next token. The following shows this path in GPT-2 (a relatively classic LLM) and MiniMax-M3 (a more recent LLM released this year; a placeholder for most other recent LLMs). As we can see, the architectures are not too different from each other, but before we get to the positional information parts, let’s focus on the orange blocks for now (the token embedding layers) and the transformer blocks above them; the other architecture differences aren’t needed for this explanation. Figure 2: Token embedding layers in GPT-2 and MiniMax-M3, highlighted in orange. (Side note: M3’s first three out of 60 feedforward layers are dense not MoEs.) A token embedding layer, shown in the figure above, maps each token ID to a vector. Position information is then added afterward. Concretely, and as we will discuss in more detail later, GPT-2 adds a position vector to each token embedding before the modified embedding enters the first transformer block. On the other hand, MiniMax-M3 adds the positional information in each transformer block using a method called RoPE inside attention. We will talk about these methods in more detail later, but first, let’s start with the big picture information explaining why explicit positional information is added or needed. 1.1 Why add positional information? Consider the following two input sentences as a simple example: “The red fox jumps over the gray fox.” “The gray fox jumps over the red fox.” Changing the word order changes which fox does the jumping, as illustrated in the following figure. Figure 3: Changing the word order changes which fox jumps. For the examples in this article, we will treat each word as one token for simplicity. (As you know, real tokenizers can split words into smaller subword tokens.) The first sentence then has eight token positions, from 0 through 7. Figure 4: Token IDs and positions in the toy sentence. # Assume the following small vocabulary. # Real vocabularies typically have 50k to 200k entries vocab = { "The": 0, "red": 1, "fox": 2, "jumps": 3, "gray": 4, "over": 5, "the": 6, } (If you are interested in how vocabularies are built, check out my tokenizer materials on GitHub .) Note that “fox”, which appears twice in the input sentence, gets the same token ID 2 in both cases. And since they both have the same token ID, irrespective of the position in the input sentence, they would get the same embedding vector from the embedding layer. (For those who are interested in more details on how this selection from the embedding table looks like, check Appendix A .) Now, let’s consider a regular self-attention mechanism with no positional encoding (and no causal mask), where every input token can attend to every other token in the input. In this scenario, swapping any tw

Ahead of AI · 15 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
56

AI models are inevitably going to keep getting better and better at all coding, tool and computer use, math, and other coding-related tasks. As a result, agents that can use software to do their work,

50.6

Hướng dẫn levie

AI models are inevitably going to keep getting better and better at all coding, tool and computer use, math, and other coding-related tasks. As a result, agents that can use software to do their work, or generate code from scratch, will improve at a faster rate than all other domains of knowledge work. On one hand that could mean that only a small subset of work can get automated. But probably what happens is that more and more work just becomes coding-shaped to adapt to what agents are good at. You can see it in the most recent examples of model releases in the past couple of months, where agents can generate code that becomes animated videos, games, 3D models for products, and so on. We also know that tool use gives agents the ability to work with much of the same software that people already work with. This helps in cyber, life sciences, finance, and a number of other domains. Our mental model traditionally has been that the models have to be able to know everything about anything to generate value, but in reality they *really* have to know how to use or generate software effectively combined with deep domain expertise to make the right judgment calls inside of workflows. When you have that powerful combination, almost everything else gets unlocked. This is why the acceleration of agents in knowledge work is only going to increase from here.

X · 15 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
57

Two very useful prompts to try with Fable 5.2:

50.4

Hướng dẫn petergyang

Two very useful prompts to try with Fable 5.2: "/doctor prompt-audit": Audits your Claude Code setup (Claude md, skills, commands) and proposes fixes. "/claude-api prompt-audit [skill name]": Audits one skill in depth. I wish Claude Code auto-suggested running these every time Anthropic releases a new model. 📌 Watch my full tutorial on how I use Fable and Opus here: https://youtu.be/fAkm3YQg73s

X · 14 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
58

Here's the full guide: how to check any benchmark in 5 minutes, and how to run an eval on your own documents - https://www.datalab.to/blog/how-to-evaluate-ocr .

48.7

Hướng dẫn vikparuchuri

Here's the full guide: how to check any benchmark in 5 minutes, and how to run an eval on your own documents - https://www.datalab.to/blog/how-to-evaluate-ocr .

X · 14 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
60

Highly recommended read.

48.7

Hướng dẫn omarsar0

Highly recommended read. My notes: I have recently started independent research on co-evolved models. My focus has been on co-evolving models and harnesses. This work focuses on memory writing specifically for continual learning. Why is this a big deal? As you work with an agent, it should get better at your tasks, but that obviously doesn't happen easily, and current external memory systems often aren't enough. This has less to do with memory systems (there are many great ones out there) and more to do with the model's awareness and judgment about when to act. To illustrate, imagine an AI agent spends a week fixing bugs in the same project. We would expect it to learn how the project works and avoid repeating mistakes. Giving it a place to save notes lets it carry that knowledge into later tasks. But choosing what belongs in those notes requires judgment. This work looks at how to train that judgment with RL. They essentially teach an agent to discover useful information while working and write memories that help it succeed on later tasks. A trained 4B parameter model reached a score similar to GPT-5.4 and Claude Opus 4.7. RIME uses one model in two roles. As the actor, it works on a task using its existing memory. As the writer, it reviews what happened and updates the notes available for the next task. The training rewards each role differently. The writer receives credit when performance on the next few tasks improves relative to performance on the recent tasks. This gives it a reason to preserve information that will help later. The comparison is an approximate training signal, since other actions and memory updates also affect those results. The actor earns a reward for completing its task and an additional reward for discoveries that reach memory. An AI judge compares the old and new notes, looking for useful additions, corrected beliefs, and concrete leads. This encourages the agent to investigate enough to learn something reusable. Rewriting the same notes in different words earns little credit. The researchers found that these abilities need to develop together. A writer depends on the information the actor discovers, and an actor depends on the knowledge the writer preserves. One big question I have here is efficiency and, obviously, how it all integrates into harnesses and workflows. I think approaches like this can save a lot of tokens and turns. While this is on a small scale, it's great to see it working already. It feels like a more feasible approach for continual learning. Frontier models are getting better at retaining knowledge, but I think efficient continual learning will still require touching the weights, which is where open-weight models will play a huge role.

X · 14 thg 11, 2026

Nguyên văn nguồn
Chi tiết nguồn
55–60 / 179
Trang 10 / 30