Text diffusion: a GPT that writes all at once

Intro

0:00 Hello and welcome back to episode fifteen, where we're going to turn our GPT into a diffusion model, which works on the whole story at each step, and writes it by unhiding a few tokens at a time. Let's begin with a quick look at where we're going to be by the end of the episode.

0:17 By the end of today, our model writes a story like this. It starts from a window of hidden tokens, and over sixty-four steps, it unhides them, a few at a time, anywhere in the story.

0:28 On the website of the series, you can find more information about the project. The code is open source on GitHub, tagged at each step of each episode, so that you can follow along, and the docs get a page on text diffusion today, with all of today's measurements. And each episode has its own page here, with its chapters and its code.

Writing all at once

0:48 Now, before we get to today's work, let's quickly go over what we did last time. We served our model as an API in the shape of OpenAI's completions, for a crowd of players, and wrote the stories for the requests that were waiting together, as a batch. Each of those stories was written the way all our models write, a token at a time, from left to right, with each token predicted from the ones before it. That's next-token prediction, and a model that writes that way is called autoregressive. Today, we're going to try another way, which is how diffusion models write: start from a window of hidden tokens, and unhide them, a few at a time, anywhere in the window, with each step seeing the story on both sides of each gap.

1:29 Diffusion for images is how most image generators work, and for text, it's quite recent. This is LLaDA, large language diffusion with masking, from Shen Nie and others at Renmin University of China, and their code on GitHub. It trains the recipe we're going to write today at eight billion parameters, from scratch, on about two point three trillion tokens, and it reports results close to Llama 3's eight billion parameter model. So the recipe works at that scale, and we're going to keep coming back to LLaDA's code, to compare it with ours, which is small enough to read from one end to the other.

Filling in what's hidden

2:05 Training starts from a window of tokens from the stories, x, and adds noise to it. The noise is a new token, the mask token, which replaces each token with a probability t, drawn anywhere between zero and one for each window. So some windows lose a few tokens, and some lose nearly all of them. The model sees the noisy window, x t, and the loss is the usual cross-entropy, but on the hidden tokens alone: how well it gets back each token that was replaced, from what's left on both sides. Each window's loss is divided by its t, which gives the windows with a few tokens hidden more weight than those with many. With that weight, the average is a bound on the negative log likelihood of the window, which the loss of next-token prediction measures directly. So we can put the two side by side, knowing that diffusion's number is an upper bound. And a t drawn anywhere from zero to one is what makes this a model that can write: BERT, from 2018, hid a fixed fifteen percent, so it didn't learn to start a story from a blank window.

3:05 The model hardly changes, so we start in model dot py, where the config gets a new field, causal, which is true by default, so the checkpoints of all our earlier episodes load as they were. Its comment says what it means: each token sees the ones before it, and for diffusion, all of them. Attention keeps the setting as self dot causal, and our flash attention kernel from episode thirteen is causal, so it's used when the model is, and otherwise, PyTorch's attention gets is causal set to the model's setting. Its comment now says that a model that isn't causal sees all the tokens in the sequence, before it and after. The weights don't change either way, so attending both ways costs no parameters. The rest goes in a new file, diffusion dot py. It imports argparse, for the commands it's going to have, os and textwrap, to wrap the stories we print, torch and its functional module, and from our own files, the data folder and load tokenizer, from data dot py, and end of text, from tokenizer dot py. Mask id is the id of our new token. It goes right after the tokenizer's ids, so it's the length of the tokenizer, and the comment says why: it's the last row of the model's embedding, and it's not in the stories, it's written in where the noise hides a token. Add noise takes a batch of windows, x, a rate for each row, and the mask id. Torch dot rand draws a number between zero and one for each token, and a token is hidden where its number is under its row's rate. We keep the first token of each window, and the comment says why: so there's a token to start from, which we're going to need in a moment. Masked fill writes the mask id where a token is hidden, and add noise returns the noisy windows and which tokens it hid. Hidden losses runs the model on the noisy windows. Then the targets, which are where training differs from next-token prediction. Our head still predicts the token after each position, and we keep it that way, as Dream 7B does: the mask at position i is filled from the output at position i minus one. So the targets are x from its second token, with minus one where the token wasn't hidden, which cross-entropy leaves out, as in fine-tuning. The comment above them says that the output at i minus one now sees the whole sequence, before it and after, so it can fill the gap from both sides. Cross-entropy, with reduction none, gives a loss per token, and it returns them per row, with zero for the tokens the model could see. Diffusion loss draws a rate per row, and where LLaDA draws each one independently, we spread them evenly across the batch, from a single random offset, as MDLM does, so each batch has rows with a few tokens hidden and rows with nearly all of them, which steadies the loss from step to step. The clamp keeps a rate from being zero, as we divide by it. Hidden losses gives each row's losses on its hidden tokens, and then each row's losses are summed, divided by its rate, and by the window's length, which puts them per token, on the scale of next-token prediction's loss, and the mean over the batch is the loss we train on. Show prints a window, with a shaded block for each hidden token, and the tokens in between decoded as text. Masking is our first command: it loads the BPE tokenizer from episode eleven, reads the first stretch of the held-out stories, and picks the first story shorter than six hundred characters, with end of text in front of it, as windows in training have it. Then, for each rate it's given, it adds noise, and prints how many tokens it hid, and the story with its blocks. Main sets up argparse with subcommands, so diffusion dot py can grow a command at a time, and for now, mask takes the rates, a tenth, half and nine tenths by default, and a seed. And tests slash test diffusion dot py, which imports math, for the log, torch, and from our own files, add noise and diffusion loss, and the GPT and its config. The mask is fifty minus one, the last of fifty ids. Tiny builds a tiny model, two blocks of thirty-two channels with rotary embeddings, causal or not, from a fixed seed, without dropout and in eval mode, so its outputs don't change between two runs of it. The first test changes a token near the end of a window, at position twenty, and compares the outputs before it, for a causal model and then a diffusion model. A causal model's outputs don't move, as it can't see ahead, and a diffusion model's do, as it attends to position twenty too. The second test adds noise to four long rows, at rates of zero, a tenth, half and nine tenths. It checks that the first token is kept, that the mask is where add noise says it hid a token, and that the tokens it kept are the originals. Then it checks that each row hides about its rate, within five percent. The third test runs our loss on random windows, with an untrained diffusion model, and checks that it's about the log of fifty. Before training, all the tokens are about as likely as each other, at any amount of noise, so the bound starts where next-token prediction's loss starts, which tells us that dividing by the rate and by the length puts it on the right scale.

8:08 So, python diffusion dot py mask, with its three rates. With a tenth of the tokens hidden, a reader would fill most of the gaps without thinking, as each has plenty of story around it. At half, about as many tokens are hidden as are left, and at nine tenths, there are a few words a line left, which is close to writing the story from scratch. The model has to learn all of these, and the ones in between.

8:32 And the three tests pass, so a diffusion model sees ahead, the noise hides what it should, and an untrained model starts at about the log of the vocabulary's size.

8:42 LLaDA's guidelines for training show how close this is to their code at scale. Starting from an autoregressive model, they get LLaDA's network by removing the causal mask from attention, as we did with a flag. And their forward process draws t for each window, hides each token with probability t, with the mask token's id, and divides each hidden token's cross-entropy by t, as our loss does. The differences between the two are small: they predict each hidden token at its own position, where our head keeps predicting the next one, and we spread the rates across the batch. And they train one percent of their windows at a random length, so the model doesn't expect a full window each time, which we can skip, as our sampler fills a full window, the length the model trained on.

Writing by unhiding

9:27 To write, diffusion dot py imports load checkpoint, to load a trained model, and pick device. Denoise is the sampler, and it runs under torch no grad, as we don't train here. It takes the model, a canvas of token ids, the mask id, the number of steps, a temperature and top k, as our generate does, and a ladder, a list for a copy of the canvas after each step, so we can watch it. It works on a clone of the canvas, so the caller's stays as it was, and above the clone, the comment says what denoise does: it writes each mask in the canvas, some of them at each step, and leaves the other tokens as they are. For each step, hidden is where the canvas still has masks, and left counts them for each row. When the sum of left is zero, all the masks are written, so we break out of the loop. The model runs on the whole canvas at each step, and there's no KV cache, as a new token changes what all the others see. Then the logits are shifted, like in training, in float32, and divided by the temperature, and the comment over them says why: position i minus one's output predicts token i, so the first token can't be hidden, and on a GPU, the logits can come in bfloat16. The mask token's logit goes to minus infinity, as the mask isn't a word to write. With top k, kth is the fiftieth largest logit at each position, and the logits below it go to minus infinity too. Softmax turns them into probabilities, and multinomial samples a guess for each position at once, as the comment above it notes. Sure is how likely the model found each guess, which gather picks out of probs, a number for each position. Pad puts back a column at the start, for the first token, and the positions that aren't hidden get a sureness of minus one, so they can't be picked. Keep is how many we're going to write at this step: what's left, divided by the steps left, rounded up, so the last step writes the rest. The comment above it says how we choose: of the hidden tokens, the surest guesses are kept, and the others stay hidden, for a fresh guess at the next step, with more of the story around them. Rank is each position's place, from the surest down, which two argsorts give us, and reveal is the hidden positions ranked under keep. Those guesses go into the canvas, and if there's a ladder, it gets a copy of the first row, and after the last step, denoise returns the canvas. Canvas for makes a window of masks, as long as the model's context, with the given ids at its start, on the model's device, and its comment says so. First story finds the story a canvas starts with. Its comment says how: the canvas starts with end of text, as a story does in training, so its story runs to the next end of text. What comes after that is the start of another story, as the windows the model trained on run on from one story into the next. Tokens cuts it short, which the training log is going to use. Write makes a story by denoising, and the comment sums it up: end of text and the prompt, then masks to the end of the window. So we fill a full window each time, the length the model saw in training, and LLaDA's trick of training on a few shorter windows isn't needed. Write puts the model in eval mode, as dropout would add noise to each guess, denoises the canvas, puts the model back as it was, and returns the first story. Fill is infilling, and its comment describes it: a story's beginning and its end, with a gap of masks between them, and after its end, the rest of the window, as in training. A next-token model can't do this directly, as it would have to write the end without seeing it. Then come two commands, and writing loads a checkpoint on the best device we have, and for each story, it makes a fresh ladder, and calls write with the prompt and the settings. With ladder, final is the last canvas, and end is where its first story ends, so we show the canvas as far as its first story goes once it's written, which the comment notes. For each step we asked for, it prints the step's number, then the canvas at that step, up to end, with show from earlier, a shaded block for each token still hidden, wrapped at ninety characters. Without ladder, it prints the story, wrapped at ninety characters too. Filling loads a model too, and for each story, it calls fill with the beginning, the end and the gap, and prints the infilled story. And in main, a loop adds the two commands, write and fill, with what they share. Each gets a checkpoint, a number of stories, with dash n for short, and the steps, sixty-four by default, which is the setting we're going to measure later, as it trades time for quality. Then the temperature and top k, eight tenths and fifty by default, as sample dot py has them, and each command runs its own function. After the loop, write gets its own arguments, a prompt, once upon a time by default, and the ladder, a list of the steps to show the canvas after, while fill gets the story's beginning and its end, and the gap, forty tokens by default. In the test, the import adds denoise, and the new test starts from three rows of thirty-two masks, and puts five random tokens at the start of each, as a prompt would be. It denoises them with our tiny, untrained model, in nine steps, keeping a ladder. It checks that all the masks are written, and that the five given tokens stay as they were. And it checks that the ladder counts down by three each step, twenty-seven tokens in nine steps, as the comment says, which pins how many tokens each step writes.

14:52 LLaDA's sampler is in generate dot py, and these lines are its low-confidence remasking. The probabilities are a softmax of the logits, and x zero p is the probability of each position's guess. Confidence is minus infinity where the canvas isn't a mask, and top k on each row picks the surest positions to write at this step. How many it picks comes from a function above, which spreads the masks evenly across the steps, as our denoise does. Theirs picks its guesses with Gumbel noise, which with a temperature of zero is the most likely token, and it can write the window in blocks, from left to right, which they call semi-autoregressive. Ours samples with a temperature and top k, like our generate.

15:33 The four tests pass, so denoising writes all the masks, in as many steps as we ask, and leaves the prompt alone.

Ten minutes on the laptop

15:41 Training needs the loss and the sampler in pretrain dot py, which imports diffusion loss and mask id, and write, as write by denoising, so that it doesn't clash with our sampler for next-token models. Next token loss is the loss we've used so far, and estimate loss takes the loss to measure, next-token loss by default. Main gets a diffusion flag, and the run's folder ends in diffusion. A diffusion model has one more token in its vocabulary, for the mask, and isn't causal, and the comment says so. Loss of picks the loss: diffusion hides some of x's tokens and learns to fill them in, while next-token prediction learns y, the tokens shifted by one. The evaluations write their sample by denoising, and show its first forty-four tokens, as the next-token model writes forty after a prompt of four. And the training step calls loss of, while the windows of stories, the optimizer and the schedule stay as they were. Pretrain DDP dot py, from episode twelve, gets these changes too, with a dash dash diffusion flag, so we can train a bigger one on rented GPUs later on.

16:45 So, ten minutes on the laptop, with rotary embeddings and the BPE tokenizer, like the next-token model from episode eleven, and dash dash diffusion. It has thirteen million seven hundred and ninety-three thousand six hundred and sixty-four parameters, three hundred and eighty-four more than that model, the embedding's row for the mask. At first, the loss is about the log of the vocabulary, and the sample is noise. It trains at about twenty-seven thousand tokens a second, and in ten minutes, it takes one thousand two hundred and sixteen steps, against one thousand two hundred and thirty-four for the next-token model in episode eleven. By the end, its held-out loss is down to about three point one one, and the samples read like the start of a story, with words in the wrong places.

17:29 The held-out loss falls from about nine point one to about three point one one per token. The dashed line is the next-token model from episode eleven, trained for ten minutes as well, at about two point one two on the held-out stories. So far, diffusion is well behind, as its number is a bound, and as it learns a harder task, where each window has a different set of tokens hidden, from a few to nearly all of them.

17:53 Diffusion dot py write, with a ladder at steps eight, sixteen, thirty-two and sixty-four. After eight steps, the words the model is surest of are in, mostly next to the prompt: there was a little, named, one day. Each step, the gaps fill in from both sides, mostly from left to right, and by step sixty-four, the window is written. This story has no end of text, so it runs to the end of the window, and it goes back to the park again and again, as a model trained for ten minutes does. Some words come out twice in a row, like play play, as two neighbours guessed in one step don't see each other's guess, so both of them can pick play.

18:32 And fill, with a beginning, a little dog named Max, and an end, where Max isn't scared of the dark any more, and forty tokens to write between them. The first story leads from one to the other, with a red ball and the park in between, and the second gets muddled, with Max four times in a few words. Both join the two ends, though, which our next-token models can't do.

Measuring it

18:55 To compare the two kinds of model, we go back to bits dot py from episode eleven, which measures bits per character on held-out stories, so that models with different tokenizers compare. It imports json, for writing the results, and hidden losses and mask id from diffusion dot py. Rates is sixteen, the slices of zero to one a diffusion model is measured at. In bits per character, nats sums the loss, and predicted counts the tokens a model predicted, while rate nats and rate hidden keep the loss and the hidden tokens at each rate. The seed is fixed, so the noise doesn't change from one run to the next. For each batch of windows, a causal model is measured as before, and adds its tokens to predicted. For a diffusion model, there's a loop over the sixteen slices, and the comment above it says what we compute: its loss is a bound, its loss on the hidden tokens at a mask rate from each of sixteen equal slices of zero to one, divided by the rate, averaged over the slices, with the first token given. So each row draws a rate within slice k, hidden losses adds the noise and scores the model, and nats gets each row's loss divided by its rate, over sixteen, which is the integral over t from the formula, a slice at a time. Rate nats and rate hidden keep what each slice saw, and predicted counts the tokens after the first. The characters are counted for the tokens predicted, and the results get the loss per token as well. For a diffusion model, they also get by rate, the loss per hidden token at each rate, from a few tokens hidden to nearly all of them, as its comment says. In main, dash dash out names a JSON file, the model column gets wider, for our longer names, and each model's results go into rows. A diffusion model's bits per character get a less than or equal in front, and the comment says why: it's at most this, as it's a bound. After the table, each diffusion model gets a bar for each rate, and with dash dash out, json dot dump writes the rows.

20:54 Bits dot py, with episode eleven's next-token model and ours. The next-token model gets about zero point seven six three bits per character, as it did in episode eleven, and the diffusion model's bound is about one point one zero nine. Below it, the loss on the hidden tokens at each rate.

21:12 This is the chart I find the most interesting today. With three percent of the tokens hidden, the diffusion model's loss on them is about one point nine two, lower than next-token prediction's two point one two, as filling a gap with the story on both sides is easier than guessing what comes next. As more is hidden, the loss climbs, to about five point four with nearly all of it gone, where there's very little left to go on. The bound averages over all of these rates, so it pays for the hard end, which next-token prediction doesn't have to learn, as it reads the story in a single order.

21:43 To measure writing, there's a new file, diffusion bench dot py, which I'm pasting in, as most of it is the plumbing of a benchmark. By denoising writes a story from once upon a time, on a canvas the length of the window, in a number of steps. By next token writes it a token at a time, with the KV cache, until end of text, to compare with. Judged scores a story with a bigger model, the judge: its loss on the story after the prompt, with the end of text, which is lower for stories more like the ones it learned from. Its perplexity is e to that loss, per token. A judge gives dull, repetitive text a low perplexity too, so distinct counts the share of a story's runs of three words that it doesn't repeat. Measure warms up, then writes the stories one at a time, timing each between two syncs, so the GPU has finished, and held out scores real stories from the held-out set that start with the prompt, as a reference. Main loads the three models, and writes sixteen stories with each setting: a token at a time, and denoising with sixteen, thirty-two, sixty-four, a hundred and twenty-eight and two hundred and fifty-six steps, each from one seed.

22:54 Then diffusion bench dot py, against episode eleven's next-token model, judged by the bigger model from episode twelve. It writes sixteen stories with each setting, one at a time, on the Mac's GPU. Real held-out stories get a perplexity of about three point two from the judge, and the next-token model's about six point four, in about four hundred and thirty milliseconds a story.

23:18 Each step runs the model once over the whole window, so the time grows with the steps, from about one hundred and twenty milliseconds at sixteen steps to about nineteen hundred at two hundred and fifty-six, where it writes about a token a step. Writing a token at a time, with the cache, takes about four hundred and thirty, so on the Mac, denoising is faster up to thirty-two steps, and about even at sixty-four. Fewer steps cost quality: at sixteen steps, about sixteen tokens are guessed at once, and the judge's perplexity is about twenty-five, coming down to about seven at a token a step, against about six for the next-token model. The judge scores repetition as likely, though, and the share of runs of three words a story doesn't repeat falls from about eighty-five percent to seventy-one as the steps go up, against ninety-six for the next-token model. So, after ten minutes on a laptop, the diffusion model's stories are less varied as well.

24:10 LLaDA's authors say this plainly in their questions and answers. At their scale, it samples slower than an autoregressive model, for three reasons: it writes a fixed length, it can't use a KV cache, and it does best when the steps equal the length of the answer. They point to work that takes these on, such as block diffusion, caches for diffusion models, and fewer steps by distillation.

From a next-token model

24:32 Training a diffusion model from scratch is the expensive way. Dream 7B, from the University of Hong Kong, started from the weights of Qwen2.5, a next-token model, and we can do that too. From autoregressive loads a next-token model's weights into a diffusion model. The two models are shaped alike, apart from the embedding, where the next-token model's is a row short, so we add the new model's row for the mask to it, and the head shares it. Load state dict puts the rest in as they are. As our head still predicts the next token, the converted weights start close to what diffusion needs. What they haven't learned yet is to use the tokens after each position, and the masks. Pretrain dot py takes dash dash init with a path, and the run's folder ends in converted. Pretrain DDP dot py takes it too, and prints where the model starts from. And the test saves a tiny next-token model, converts it, and checks that the embedding and a block's weights are the next-token model's, and that the head still shares the embedding.

25:36 Then five minutes, starting from episode eleven's next-token model. Before a single step, its diffusion loss is about five point zero, where a model from scratch starts at about nine point one, so much of what it knew carries over. In five minutes of training, its held-out loss gets to about two point eight zero. That's about one point zero zero nine bits per character, against one point one zero nine for ten minutes from scratch, although it had the next-token model's ten minutes before them.

On four GPUs

26:03 For a fair test, I've rented the kind of machine episode twelve trained on, four RTX 4090s in Norway, for under three dollars an hour, and given it the name gpu for ssh, as we did in episode thirteen. Nvidia smi shows the four GPUs, idle, with twenty-four gigabytes each.

26:22 Remote dot sh, from episode twelve, copies the code and the BPE merges, builds the C++ tokenizer, and downloads all of TinyStories and tokenizes it, on all the machine's cores. And scp uploads the model episode twelve trained on four GPUs, which we're going to convert.

26:47 Then the run episode twelve made, thirty minutes on four GPUs, with GPT-2 small's shape, five hundred and twelve tokens of context, ninety-six windows a GPU a step and torch compile, with dash dash diffusion. It runs in the background, with nohup, so it carries on if the connection drops, and tail follows its log. During the first steps, PyTorch prints a warning from each GPU about running out of memory. Its allocator ran short once, freed the memory it was holding in its cache and tried again, so the run carries on. The model has about ninety-one million parameters, and it trains at about four hundred and fifty thousand tokens a second, against about four hundred and seventy thousand for the next-token model, as attention both ways skips no blocks of the scores, and the loss keeps a value per token. In thirty minutes, it sees about seven hundred and sixty million tokens, and its held-out loss comes down to about one point six five.

27:46 Side by side with episode twelve's run, step for step, the next-token model ends at about one point zero seven, and the diffusion model at about one point six five. The next-token model was under that by step five hundred. Diffusion's number is a bound, so its true loss is somewhat lower, by an amount we can't measure directly. Even so, with this much compute, next-token prediction learns the stories a good deal faster, which is what papers on scaling these models find as well: one from LLaDA's authors found that, to write as well as a next-token model, a masked diffusion model needed about sixteen times the pretraining.

28:23 Then fifteen minutes from episode twelve's model, with dash dash init. Before its first step, its diffusion loss is about five point six, against nine point one for a model from scratch. Its starting loss is well above where it was at next-token prediction, as it hasn't yet learned to use the tokens after each position, or the masks. As before, each GPU prints the warning about memory during the first steps, and the run carries on. And the first evaluation shows how fast that changes: about two point zero three at step two hundred and fifty, where the model from scratch was at two point eight two. After fifteen minutes, its held-out loss is about one point six two.

29:10 Step for step, the converted model stays ahead, and in fifteen minutes it ends a little under where thirty minutes from scratch did. It had thirty minutes of next-token training before them, so in total it isn't cheaper, but a next-token model is something you often have already, and then converting it halves the time to a diffusion model this good.

29:30 Bits dot py, on the rented machine, for the three of them: the next-token model gets about zero point three eight six bits per character, as in episode twelve, diffusion from scratch about zero point five eight eight, and the converted model about zero point five eight five. Below them, the loss by rate looks like the laptop's, with about zero point three eight per hidden token at three percent, far under the next-token model's one point zero seven.

29:57 With thirty minutes on four GPUs for each model, diffusion's bound is about fifty-two percent above next-token prediction's, and the converted model, in fifteen minutes, ends about where thirty from scratch did.

30:09 And we time the stories on a single 4090, with the converted model. Here the judge is the next-token model, which scores its own stories as more likely than real ones, so we compare diffusion with the real held-out stories, at a perplexity of about three point three. A token at a time, with the cache, a story takes about eleven hundred milliseconds. Denoising takes about one hundred and twenty at sixteen steps, with a perplexity of about twenty-three, about nine hundred and fifty at one hundred and twenty-eight, close to the next-token model's time, at about five point one, and it takes two hundred and fifty-six steps, about nineteen hundred milliseconds, to write stories the judge finds as likely as real ones.

30:51 Scp copies the two models back to the Mac, with the runs' logs and the measurements, and vastai destroy instance gives the machine back, which cost about two point nine dollars in all.

31:21 Back on the Mac, the diffusion model trained from scratch on four GPUs, writing a story in sixty-four steps. At steps eight and sixteen, most of the story is still hidden after its first line: the other tokens the model was surest of are after the story's end, the end of text and the start of the next story, which we don't show. From there, it writes mostly from left to right, next to what it's already written, and by step sixty-four, it's a story about helping to bake a cake, with a few slips, and an ending, they lived happily ever after.

31:53 And the converted model, filling in the middle of Max's story. In the first, Max is a fearful dog, scared of the dark, his mom gives him a big hug, and it leads into the given end with the words from that day on. In the second, Max makes friends with a kind girl, with a few slips on the way, and both stories reach the end they were given.

Wrap-up

32:13 Last, the docs get a page on text diffusion, with the loss and its bound, the sampler, next to LLaDA's, and all of today's measurements, on the laptop and on four GPUs. It goes in the list on the docs' home page, and at the end of the navigation. And the README has the commands to train, write, fill and measure a diffusion model.

32:33 So, today we turned our GPT into a masked diffusion model, with attention both ways, a mask token, and a loss on the tokens hidden at any rate from zero to one, which is a bound on the negative log likelihood. It writes by unhiding the tokens it's surest of, a few at a time, the way LLaDA does at eight billion parameters, and fewer steps are faster, at a cost in quality. With thirty minutes on four GPUs, like episode twelve's model, its bound ended at about zero point five eight eight bits per character, against zero point three eight six for next-token prediction, and converting episode twelve's model got there in half the time. And it fills in the middle of a story from its beginning and its end, which our next-token models can't do.

33:18 Let me know in the comments whether you'd like more on diffusion models, such as writing in blocks or fine-tuning one to follow instructions, and which other ideas from recent papers you'd like us to try on mvGPT. And if you'd like to follow along, subscribe to the channel. I hope you find this useful, and thanks for watching.