RMSNorm, SwiGLU, QK-norm: a modern GPT block

Intro

0:00 Hello and welcome back to episode seventeen, where we're going to bring our model's transformer block up to date, with the changes that recent open models have made to it. That means RMSNorm, SwiGLU, a norm on the queries and keys, and no biases. Let's begin with a quick look at where we're going to be by the end of the episode.

0:20 By the end of today, our model can be built with a modern transformer block, and summary dot py prints it as PyTorch sees it. Each layer norm is an RMSNorm, the MLP has a gate projection next to its up and down projections, and attention has a norm for its queries and one for its keys. None of the linear layers has a bias, and the model has fourteen million four hundred and sixty-two thousand five hundred and ninety-two parameters, about as many as before.

0:48 On the website of the series, you can find more information about the project. The code is open source on GitHub, tagged at each step of each episode, so that you can follow along, and the docs get a page on the modern transformer block today. And each episode has a page here as well, with its chapters and its code.

1:07 Now, before we get to today's work, let's quickly go over what we did last time. We trained our model with GRPO, where it wrote eight stories for each instruction, and each story was scored against its group's mean and spread, in place of PPO's critic. With PPO's KL weight, the share of the requested words its stories used went from about twenty-seven percent to fifty, against about forty-five for PPO. That's a single run of each, so I'd call the two close at our scale. With no penalty at all, the stories used about fifty-eight percent of the words, but eight percent of them ended on their own, against forty-eight for the fine-tuned model.

1:43 Today, we leave training aside, and go back to the model's architecture. In episode one, our transformer block followed GPT-2's, with a layer norm before attention and another before the MLP, and GELU in the MLP. Each of its linear layers and layer norms has a bias, as in GPT-2. In episode ten, rotary embeddings became an option in place of our table of positions. We're going to change the rest of the block, starting with RMSNorm, a simpler layer norm. Then comes SwiGLU, an MLP with a gate, and then QK-norm, a norm on the queries and keys in attention, which keeps the attention scores from growing with the weights. And last, we drop the biases from the linear layers and the norms. RMSNorm, SwiGLU and QK-norm each get a new field in the config, false by default, so the checkpoints we've trained so far load as before. Bias has been a field since episode one, and a flag, dash dash modern, turns all four changes on together. We won't train a model today, so we're going to write the code and its tests, and at the end, count what the changes do to the parameters. All right, let's get started with where these changes come from.

Where the changes come from

2:55 Most of this recipe is in LLaMA, Open and Efficient Foundation Language Models, by Hugo Touvron and others at Meta AI, from 2023. Its models go from seven to sixty-five billion parameters. Section two point two lists how their architecture differs from the original transformer, with where they took each change from in brackets. The first is pre-normalisation, from GPT-3, which normalises the input of each sublayer to make training more stable, as GPT-2 did, and as our model does. For it, they use RMSNorm, introduced by Zhang and Sennrich in 2019. Then SwiGLU, introduced by Shazeer and used in PaLM, takes the place of the ReLU in the MLP, to improve the model. They make the MLP two thirds of four d wide, where PaLM used four d, with d the embedding size. And rotary embeddings take the place of the absolute positions, which our model has had as an option since episode ten.

RMSNorm

3:50 RMSNorm comes from Root Mean Square Layer Normalization, by Biao Zhang and Rico Sennrich at the University of Edinburgh, from 2019. Their abstract says that LayerNorm helps a network train, but that it costs time, and their hypothesis is that one of its two steps, the re-centring, isn't needed. RMSNorm keeps the other step, re-scaling by the root mean square. In their experiments, it did about as well as LayerNorm, and took seven to sixty-four percent less time, depending on the model. Equation four is all of it: each of the inputs is divided by the root mean square of all of them, and multiplied by its gain, g, which the model learns.

4:34 A layer norm works on each token on its own, on its three hundred and eighty-four channels, which the formula calls n. It takes their mean, mu, and their variance, sigma squared, which is the mean of their squared distances from mu. It subtracts the mean and divides by the square root of the variance, the standard deviation, so the channels come out with a mean of zero and a variance of one. Epsilon is a small number that keeps the division safe when the channels are all about equal. Then each channel is multiplied by its gain, g, and gets its bias, b, added, two numbers that the model learns for each channel. RMSNorm skips both the mean and the bias, and divides each channel by the root mean square of the channels, the square root of the mean of their squares. Then each channel is multiplied by its gain, which starts at one. So, before the gain, the channels come out with a root mean square of one, and their mean isn't moved to zero. When the channels' mean is zero, their variance is the mean of their squares, so the two norms divide by equal amounts. And RMSNorm computes one statistic instead of two, with half the parameters.

RMSNorm in code

5:41 In model dot py, the config gets a new field, rmsnorm, false by default, and its comment says what it does: RMSNorm in place of LayerNorm, with each token's channels rescaled, and not centred. As it's false by default, the model we've had so far is built as before, and so is each checkpoint we've saved, as their configs don't have the field. Then the class RMSNorm, written out by hand, the way we wrote GELU in episode one, so that the formula is there in our code to read. PyTorch has it as well, as n n dot RMSNorm, and as rms norm in functional, which we could use, and which our test is going to check ours against. Its init takes the size, the number of channels it normalises, and eps, one times ten to the minus six, the small number from the formula that keeps us from dividing by zero. It calls the parent's init, and keeps eps for the forward pass. Weight is the gain, a tensor of ones, one per channel, wrapped in n n dot Parameter. A Parameter is a tensor the module trains, so the optimiser finds it, like LoRA's A and B in episode nine. Above it, the comment says it: a gain per channel, starting at one, and no bias. In forward, x32 is x, converted to float32, and the comment over it gives the reason: in float32, as squaring in bfloat16 loses precision, and then back to x's type. bfloat16 is the sixteen-bit format, with fewer digits, that our training on NVIDIA GPUs runs in, from episode twelve, and each square in it would be rounded to two or three significant digits. Meta's code for LLaMA also converts to float32 before this arithmetic. Y is the formula from the paper, where pow two squares each channel, and mean, with dim minus one, averages over the last dimension, the channels, so each token gets its own mean square. Keep dim keeps that dimension, with a size of one, so the result lines up with the token's channels when it divides them. Plus eps, and torch dot rsqrt, the reciprocal of the square root, in a single operation, which multiplies x32, so the division in the formula becomes a multiplication by the reciprocal. So RMSNorm needs a reciprocal for each position, then a multiplication per number, and no more arithmetic than that. The comment over it: each token's channels divided by their root mean square, which brings it to one, with eps against dividing by zero. It returns y, back in x's type with type as, times the gains. Norm layer takes the config and a size, and returns an RMSNorm when the config asks for one, and otherwise a layer norm, with or without its bias, as before. Its comment holds the difference between the two: LayerNorm subtracts the mean of a token's channels and divides by their spread, while RMSNorm skips the mean. PyTorch's layer norm takes a bias argument, so with bias false, it keeps its gains and drops its bias, as it has since episode one. In the block, the attn norm now comes from norm layer, given the embedding size, three hundred and eighty-four channels, like the layer norm it replaces. The MLP norm, before the MLP, comes from it as well, with that size, and these two are where each sublayer reads a normalised copy of the residual stream. And further down, in the model, so does the final norm, after the blocks, which the head reads. With rmsnorm false, norm layer gives back the layer norms we had, with their weights under the names they had, so a checkpoint saved before today loads into them as it did. And with rmsnorm true, there's no bias to load, and a gain of three hundred and eighty-four numbers for each norm. Then a new test file, tests slash test block dot py, for pytest, which runs each function whose name starts with test, as in episode thirteen. It imports torch, functional as F, and our RMSNorm. The test's name says what it checks: RMSNorm matches PyTorch's, and brings each token to a root mean square of one. With a fixed seed, x is two sequences of five tokens, with three hundred and eighty-four channels of random numbers, times three, plus one. So the channels have a mean of about one, and a version that subtracted the mean by mistake would fail. We normalise x into y, and the first assert checks that ours is within one in a hundred thousand of PyTorch's rms norm, given our gains and our eps. The second checks the root mean square of each token's channels, which should be one, to within one in ten thousand. Its comment gives the reason: with its gains at one, each token's channels come out with a root mean square of one.

10:13 We run the test file with pytest, where dash v lists each test by its name. The RMSNorm test passes, so ours matches PyTorch's, and brings each token's channels to a root mean square of one.

SwiGLU

10:26 SwiGLU comes from GLU Variants Improve Transformer, by Noam Shazeer at Google, from 2020. Its abstract describes gated linear units, or GLUs, which multiply two linear projections of their input, element by element, after passing one of them through a sigmoid. Other functions can take the sigmoid's place, and the paper tries them in the transformer's feed-forward layers, which is what we call the MLP. Equation six lists them, and the last line is SwiGLU, where x times W goes through Swish and multiplies x times V, before W two projects the result back down. As in the T5 code they started from, there are no biases. These layers have three weight matrices instead of two, so to keep the number of parameters and the computation where they were, the hidden layer is two thirds as wide. Table one has the held-out loss after about half a million steps, which the paper calls log-perplexity. SwiGLU's is about one point six four, against about one point six eight for ReLU and GELU, and the variant with GELU in place of Swish is a little lower still. And the conclusion offers no explanation for why these layers work, and puts their success down, as all else, to divine benevolence.

11:42 Our MLP, from episode one, takes each token's three hundred and eighty-four channels up to fifteen hundred and thirty-six with the up projection. It applies GELU to each of them, and the down projection brings them back down. SwiGLU has two projections going up, and one of them goes through SiLU. The result multiplies the other projection, channel by channel, which is what the circle with a dot stands for. SiLU, as PyTorch calls it, is Swish with a beta of one, which is x times the sigmoid of x. The sigmoid squashes any number into the range from zero to one. For large positive x, SiLU gives about x, for large negative x, about zero, and in between, it's a smooth curve, a lot like GELU's. With the multiplication, the SiLU side works as a gate. For each channel of each token, it scales the up projection by an amount the model computes from the input, with weights of its own. With three matrices, the hidden layer goes from four times the embedding size to eight thirds of it, two thirds as wide as before. For our three hundred and eighty-four, that's a thousand and twenty-four channels, in place of fifteen hundred and thirty-six. So GELU's two matrices of three hundred and eighty-four by fifteen hundred and thirty-six hold about one point one eight million weights. SwiGLU's three, of three hundred and eighty-four by a thousand and twenty-four, hold as many.

SwiGLU in code

13:06 For SwiGLU, the config gets swiglu, false by default, with its comment: a gated MLP, SwiGLU, in place of GELU's. Next to GELU, we write SiLU out too, and it returns x times torch dot sigmoid of x. PyTorch has it as silu in functional, and as with GELU, writing it out keeps the formula in our code. Its comment says how it behaves: x times its sigmoid, close to x when x is large, close to zero when it's very negative, and smooth in between. In the MLP, hidden is the width of the layer in the middle. With SwiGLU, it's eight thirds of the embedding size, rounded up to a multiple of sixty-four with math dot ceil, which rounds a number up to the next whole one. Without SwiGLU, it's four times the embedding size, as before. The comment over it says why: SwiGLU has three matrices to GELU's two, so at eight thirds of the embedding size wide, it has as many weights. It goes on: rounded up to a multiple of sixty-four, a size GPUs handle well, a thousand and twenty-four for our three hundred and eighty-four. Without the rounding, an embedding size of five hundred and twelve would ask for a width of thirteen hundred and sixty-five and a third, which no layer can have. Gate proj is the new projection up, from the embedding size to hidden, built for SwiGLU, and none otherwise, so the original MLP has no extra weights. Its comment: the gate, a second projection up, which scales each of up proj's channels on the way through. These names, gate proj, up proj and down proj, are the ones Hugging Face's code for Qwen3 uses. The up projection and the down projection now go to hidden and back, instead of four times the embedding size. In forward, if there's no gate, the MLP runs as before, with the up projection, GELU, the down projection and dropout. Otherwise, gate proj's output goes through SiLU and multiplies up proj's, channel by channel, and down proj brings the product back to the embedding size, before dropout, which is the formula from the paper. In the test file, we import the MLP and the config too. The new test checks that SwiGLU has as many weights as GELU's MLP. For swiglu false and then true, it builds an MLP with no biases, as with biases, SwiGLU's count would be five hundred and twelve higher. Numel is the number of numbers in a tensor, and we sum it over the MLP's parameters and append the count. The comment over the assert does the arithmetic: GELU's two matrices of three hundred and eighty-four by fifteen hundred and thirty-six, and SwiGLU's three of three hundred and eighty-four by a thousand and twenty-four. Both come to one million, one hundred and seventy-nine thousand, six hundred and forty-eight weights. And the assert checks that both counts come to that.

15:47 Pytest again, and both tests pass, the RMSNorm one from before and the new one, so without biases, SwiGLU's MLP has as many weights as GELU's.

QK-norm

15:58 Normalising the queries and keys goes back to Query-Key Normalization for Transformers, by Alex Henry and others, from 2020, for translating languages with little training data. Their abstract says that the technique makes the softmax in attention less prone to saturating, which is when nearly all of a query's weight goes to a single key. They normalise each query and each key along the head's channels before multiplying them. Then they scale the result up by a parameter the model learns, instead of dividing by the square root of the dimension. Section three shows the problem, with an example of the softmax, which responds to the differences between its inputs. So seven hundred and sixty, seven hundred and fifty-two and seven hundred and fifty give the weights that twelve, four and two give, about ninety-nine point nine six percent on the first. A dot product has no upper bound, so as the scores grow, a difference that's small next to them can put nearly all the weight on a single key. Equation five is their attention, with the normalised queries times the normalised keys. Each score is then the cosine of the angle between a query and a key, between minus one and one, times g, the learned scale.

17:06 The version recent models cite comes from Scaling Vision Transformers to Twenty-Two Billion Parameters, by Mostafa Dehghani and others at Google Research, from 2023. As they scaled up their vision transformer, they saw the training loss diverge after a few thousand steps, in models of around eight billion parameters. They traced it to extremely large attention logits, the scores that go into attention's softmax. The logits made the attention weights almost one-hot, with nearly all of a query's weight on a single key. To fix it, they applied a layer norm to the queries and the keys before the dot product, and kept the division by the square root of d. Figure one shows an eight-billion-parameter model with and without it. Without the norm, in red, the loss turns back up after about two thousand steps, and the largest attention weight goes to one. With it, in green, the loss keeps going down. Their appendix says that without the norm, the logits grew past fifty thousand. They also removed the biases of the query, key and value projections, and ran their layer norms without a bias or centring, citing Zhang and Sennrich. Dropping those made better use of their accelerators, by three percent, with no loss in quality. They kept the biases in the MLP, though, as they found them better for quality, at no cost in speed.

18:24 A model doesn't have to be that large to run into it. Small-scale proxies for large-scale Transformer training instabilities, by Mitchell Wortsman and others at Google DeepMind, from 2023, studies two instabilities seen in large models. The growth of the attention logits is one of the two. Their abstract says that both show up in small models too, when they're trained with high learning rates, and that the fixes used at scale work there as well. Figure two shows a model of nine point four million parameters, smaller than ours, trained with a learning rate of zero point one. We pretrain with six times ten to the minus four. Without qk-layernorm, in orange, the largest attention logit climbs to about a million, and the loss jumps up. With it, in blue, the logit stays flat, and the loss stays down. Their hypothesis is that the logits grow with the square of the size of the weights, as the queries and the keys both grow with them, and a logit multiplies the two.

19:21 In attention, a query's score against a key is their dot product, q, k transposed, divided by the square root of d k, the sixty-four channels of a head. That score is what these papers call an attention logit, and it has no upper bound. If training makes the query and key projections larger, the query and the key both grow, so the score grows with the square of their size. QK-norm runs each head's query and each head's key through RMSNorms of their own, one for the queries and one for the keys, over the head's sixty-four channels, before the dot product. Dehghani and others used a layer norm, and we use RMSNorm, like OLMo 2, from the Allen Institute for AI. Like them, we keep the division by the square root of d k. With its gains at one, an RMSNorm gives a head's query a root mean square of one. So the query's length is the square root of sixty-four, which is eight, and the key's is eight too. A dot product is at most the product of the two lengths, so the score stays between minus eight and eight, however large the projections' weights get. And we normalise before rotary embeddings turn the queries and keys, and as a rotation keeps a vector's length, the range holds after it too. The gains are learned, so the model can still make its attention sharper, but scaling up the query and key projections no longer changes the scores.

QK-norm in code

20:42 QK-norm is the third field in the config, qk norm, false by default, and its comment says where it goes: RMSNorm on each head's queries and keys, before attention compares them. In attention's init, head size is the embedding size over the number of heads, sixty-four. Q norm is an RMSNorm over a head's sixty-four channels, when the config asks for QK-norm, and none otherwise, and k norm is a second one, for the keys. The comment over them says what they're for: queries and keys rescaled to a root mean square of one, so the scores don't grow with their projections' weights. Each has sixty-four gains, which the six heads share, as in Hugging Face's code for Qwen3, while OLMo 2's normalises all the heads' channels together. In forward, the queries and keys have just been split into heads, with view and transpose, so their last dimension is a head's sixty-four channels, which is the dimension RMSNorm works on. If there's a q norm, q and k go through their norms. That happens before rotary embeddings turn them, as in Qwen3's code, and before the KV cache keeps the keys. So the cache holds keys that are normalised and rotated, and a new token's query meets them as it would without the cache. In the test file, we import the attention too. The test's name says what it checks: QK-norm keeps attention from growing with the weights. With a fixed seed, x is two sequences of ten tokens of random channels. For qk norm false and then true, we build an attention without biases, as scaling the weights alone wouldn't scale a biased projection. It has no dropout either, so that two runs give identical outputs, and before is its output. Then, with torch dot no grad, as PyTorch won't let us change a weight in place while it tracks its gradients, we multiply the query projection's weights by ten, and the key projection's too. The assert compares the output now with before: within one in ten thousand with QK-norm, and changed without it. The comment over it says why: ten times the weights make the scores a hundred times larger, unless the queries and keys are normalised.

22:48 And the three tests pass, so with QK-norm, ten times the query and key weights leave the output as it was, and without it, they change it.

No biases

22:57 For dropping the biases, later papers cite PaLM, Scaling Language Modeling with Pathways, by Aakanksha Chowdhery and others at Google Research, from 2022. Its largest model has five hundred and forty billion parameters. Section two lists the changes they made to the transformer, one by one. SwiGLU is the first, and they note that it takes three matrix multiplications in the MLP instead of two, while Shazeer's experiments matched the compute. Further down, there are no biases in any of the dense layers, which are our linear layers, or in the layer norms. They found that this made training more stable for large models.

In recent models

23:35 Recent open models use these changes together, such as Qwen3. The Qwen3 Technical Report, by the Qwen team, from 2025, describes models from six hundred million parameters to two hundred and thirty-five billion. Its architecture section lists SwiGLU, rotary embeddings and RMSNorm before each sublayer. Grouped query attention is there too, where several query heads share a key and value head, which we haven't added. And for Qwen3, they removed the biases that Qwen2 had on its query, key and value projections, and added QK-norm, citing Dehghani and others, to keep training stable.

24:12 Andrej Karpathy's nanochat, his small ChatGPT, makes most of these choices too, and the docstring of its model lists them. It has QK-norm, its RMSNorm has no learnable parameters at all, not even the gains, and its linear layers have no bias. Its MLP goes another way, four times as wide as the embedding size, with ReLU squared, the square of ReLU's output, in place of SwiGLU, so it keeps two matrices.

One flag for all four

24:38 With all three in the config, model dot py gets MODERN, a dictionary of the four settings, with rmsnorm, swiglu and qk norm on, and bias off. Bias has been a field since episode one, and in Andrej Karpathy's nanoGPT, the comment on its own bias field says that with biases, the model is like GPT-2, and without them, a bit better and faster. The comment over MODERN says where it comes from: RMSNorm, SwiGLU and no biases, as in LLaMA, and QK-norm, as in OLMo 2 and Qwen3, while RoPE, which they use too, has a flag of its own. RMSNorm gets extra repr, which is what PyTorch prints in the brackets after a module's name when it prints the model. It returns the number of gains, numel, and eps, so the printed model shows each RMSNorm's size, and the comment says so: what printing the model shows in the brackets, as LayerNorm shows its size. In pretrain dot py, we import MODERN, and main takes modern, false by default. The run folder gets dash modern in its name, after dash rope, so a run with the modern block, such as pretrain dash rope dash modern, keeps the original block's checkpoints as they are. Each kind of model has had a run folder of its own since episode ten, and dash modern goes after dash rope and dash BPE, before the names of the diffusion models from episode fifteen. The config unpacks MODERN into its arguments with the two stars, when modern is on, and an empty dictionary when it's off, which leaves the config as it was. The comment on weight decay changes a few words, as the gains it leaves alone now include RMSNorm's: weight decay pulls the weight matrices towards zero, but not the biases and the norms' gains. The optimiser tells them apart by their dimensions, as gains and biases have one. And at the bottom, dash dash modern on the command line turns it on, as dash dash rope does for rotary embeddings. Pretrain DDP dot py, from episode twelve, imports MODERN, and gets a dash dash modern flag, whose help says what it turns on: RMSNorm, SwiGLU, QK-norm and no biases, model dot py's MODERN. The name flag has no default now, and its help gives the rule: the run's folder, by default pretrain DDP, with dash modern after it for dash dash modern. The config unpacks MODERN, as in pretrain dot py. Name is the name given on the command line, or else pretrain DDP, with dash modern for the modern block. The comment over it says why: a run with the modern block gets a folder of its own, so it doesn't overwrite pretrain DDP. The run's folder and its TensorBoard log are named after it. Summary dot py, from episode one, imports sys, for the words on the command line, and MODERN. The config reads two flags from sys dot argv, the list of those words, and the comment over it names them. Dash dash rope is for rotary position embeddings, and dash dash modern for the modern block, with RMSNorm, SwiGLU, QK-norm and no biases. With rotary embeddings, the model has no table of positions, so positions is an empty list, and otherwise it holds embed positions. The comment over it puts that in its own words: with rotary embeddings, there's no table of positions to watch. Watch, the layers whose output shapes we print, takes positions from there, as without it, summary dot py would look for a table the model doesn't have. Last, three tests of the whole model, which import GPT, MODERN and the KV cache too. The first checks that the KV cache matches running the whole sequence with the modern block. A small config keeps it fast: a vocabulary of fifty, two blocks of two heads, thirty-two channels, no dropout, rotary embeddings and MODERN. The model goes into evaluation mode, and idx is two sequences of twelve random tokens. Full is the logits of all twelve at once. Then a new cache, and steps holds the logits of each call: the first eight tokens at once, and then the other four, one call each, with the cache keeping the keys and values in between. Its comment gives the reason: the first eight tokens at once, then one at a time, as generate does. Cat joins the steps' logits along the tokens, and the assert checks that they're within one in a hundred thousand of the full sequence's. With QK-norm before the rotation and the cache, the keys the cache keeps have to be the ones a full run computes, and this test checks that they are. The next test checks that a config saved before these options builds the original block. Old is a config the way a checkpoint from before this episode stores it. It has the fields our config had until today: the vocabulary, the context window, six blocks of six heads, the embedding size, dropout, bias, rope and causal. The comment over it says what it stands for: a checkpoint's config from before this episode, without the new fields. So the three new fields take their defaults, which are all false, and the model built from it has a layer norm at the end and no gate proj in its MLP. It has fourteen million, five hundred and eighty-five thousand, eight hundred and fifty-six parameters, the count from episode one. And the last test checks that the modern block drops the biases and adds QK-norm's gains. Original counts the parameters with rotary embeddings, and modern with MODERN as well. The comment does the arithmetic: four thousand, two hundred and twenty-four biases per block go, and three hundred and eighty-four in the final norm, while QK-norm adds a hundred and twenty-eight gains per block. So the assert checks that the difference is six times four thousand, two hundred and twenty-four, plus three hundred and eighty-four, minus six times a hundred and twenty-eight, which is twenty-four thousand, nine hundred and sixty.

30:13 Let's print both blocks with summary dot py, first the original, with dash dash rope, so that there's no table of positions. Its norms are layer norms with a bias, the linear layers in its blocks have one too, and in all, it has fourteen million four hundred and eighty-seven thousand five hundred and fifty-two parameters. Then with dash dash modern as well, where each layer norm is now an RMSNorm, with its size and its eps in the brackets, from extra repr. Each attention has a q norm and a k norm, over a head's channels. The MLP has a gate proj next to its up and down projections, and none of the linear layers has a bias. Over the six blocks, each of the MLP's three projections holds two million three hundred and fifty-nine thousand two hundred and ninety-six parameters, where the two before held about three million five hundred and fifty thousand each, with their biases. In all, the modern model has fourteen million four hundred and sixty-two thousand five hundred and ninety-two parameters, which is twenty-four thousand nine hundred and sixty fewer.

31:17 And all six tests pass, so RMSNorm matches PyTorch's, SwiGLU has GELU's number of weights, and QK-norm keeps attention from growing with the query and key weights. The KV cache matches a full run with the modern block, a config from before builds the original block, and the modern block has twenty-four thousand nine hundred and sixty fewer parameters.

Counting the parameters

31:37 So, let's count the parameters of a block, part by part, before and after. Each of the two layer norms has a gain and a bias for each of the three hundred and eighty-four channels, and an RMSNorm has the gains and no bias. Attention has four projections, for the query, key, value and output, each a matrix of three hundred and eighty-four by three hundred and eighty-four with a bias of three hundred and eighty-four. The modern block keeps the four matrices without the biases, and QK-norm adds sixty-four gains for the queries and sixty-four for the keys, shared by the six heads. The MLP had two matrices, up to one thousand five hundred and thirty-six channels and back down, with a bias for each. SwiGLU has three matrices of three hundred and eighty-four by one thousand and twenty-four, with as many weights as GELU's two. In all, a block goes from one million seven hundred and seventy-four thousand four hundred and sixty-four parameters to one million seven hundred and seventy thousand three hundred and sixty-eight. That's four thousand and ninety-six fewer, as four thousand two hundred and twenty-four biases go, and QK-norm adds one hundred and twenty-eight gains.

32:44 For the whole model, with rotary embeddings, as in the episode on RoPE, there's no table of positions. The token embeddings are ten thousand rows of three hundred and eighty-four, which the head shares, so it adds no parameters of its own. Then come the six blocks and the final norm, for fourteen million four hundred and eighty-seven thousand five hundred and fifty-two parameters in all, the size of the model we pretrained with rotary embeddings. With the modern block, it's six blocks of one million seven hundred and seventy thousand three hundred and sixty-eight and a final RMSNorm with three hundred and eighty-four gains, for fourteen million four hundred and sixty-two thousand five hundred and ninety-two. That's twenty-four thousand nine hundred and sixty fewer, under a fifth of a percent of the model, so the modern block is about as large as the old one. With learned positions instead, both totals gain the ninety-eight thousand three hundred and four parameters of the table of positions.

What's real, and what's scaled down

33:38 What's real is the block: RMSNorm, SwiGLU, QK-norm and no biases, with rotary embeddings, as these papers describe them, and as Hugging Face's code for Qwen3 has them. What's scaled down is the rest, starting with the size: Qwen3's models go from six hundred million parameters to two hundred and thirty-five billion, pretrained on thirty-six trillion tokens. Ours has about fourteen million five hundred thousand, and we haven't trained it with the new block. The reasons the papers give were measured on their own models: RMSNorm's speed on their GPUs, SwiGLU's held-out loss in Shazeer's experiments, and QK-norm against the divergence of an eight-billion-parameter model. At our size, the tests check what each change does, and whether it makes a model like ours better is a question for a training run. And those models change more than we did today, such as grouped query attention in Qwen3, or PaLM's parallel layers, which we've left out.

34:37 Last, the docs get a page on the modern transformer block, with each change and the paper it comes from, and how model dot py writes it. It has the flags, the tests, and the parameter count of a block and of the model, before and after. It goes in the list on the docs' home page, after GRPO, and at the end of the navigation. And the README has the commands to print the modern block, to pretrain with it, and to run its tests.

Wrap-up

35:04 So, today we brought our transformer block up to date, one change at a time. RMSNorm took the place of the layer norms, dividing each token's channels by their root mean square, with a gain and no bias. SwiGLU gave the MLP a gate, with three matrices and a hidden layer eight thirds of the embedding size, for as many weights as GELU's two. And QK-norm put an RMSNorm on each head's queries and keys, so that the attention scores don't grow with the size of the weights. Then we dropped the biases, and dash dash modern turns all four changes on, while the checkpoints we've trained so far load with the original block, as our tests check. In all, the modern model has twenty-four thousand nine hundred and sixty fewer parameters, under a fifth of a percent.

35:52 Let me know in the comments which of these four changes you'd expect to matter most for a model of our size, and which other parts of recent models you'd like us to look at. And if you'd like to follow along, subscribe to the channel. I hope you find this useful, and thanks for watching.