Pretraining on the web: a GPT overnight

Intro

0:00 Hello and welcome back to episode nineteen, where we're going to train our model on pages from the web, for a whole night, on one GPU that we rent for under four dollars. Let's begin with a quick look at where we're going to be by the end of the episode.

0:14 By the end of today, our model has about one hundred and ten million parameters, pretrained overnight on about three billion six hundred million tokens of educational web pages, and from the prompt photosynthesis is, it writes this. It reads like a page on gardening, from keeping plants alive to sulphate poisoning, in sentences that hold together, with facts that often don't.

0:40 On the website of the series, you can find more information about the project. The code is open source on GitHub, tagged at each step of each episode, so that you can follow along, and the docs get a page on pretraining on the web today. And each episode has a page here as well, with its chapters and its code.

1:00 Now, before we get to today's work, let's quickly go over what we did last time. We gave our model grouped-query attention, where the query heads share their keys and values in groups. With two heads of keys and values, the KV cache takes a third of the memory it did. Up to now, though, our models have learned from TinyStories, short stories written with the words a three-year-old knows. Episode twelve's model, of about ninety-one million parameters, writes them well, and stories are about all it can write. So today, we train a model of that size on text from the web, which covers far more of what people write about.

1:39 Today, we start with FineWeb-Edu, a dataset of web pages that a classifier scored as educational, and download about eleven gigabytes of it. Our BPE merges were learned on stories, so the web gets merges of its own. We work out how many tokens a night on one GPU can read, and whether that's enough for a model of about a hundred and ten million parameters. Pretraining gets three new options: the web's data, gradient accumulation, for a bigger batch than fits at once, and resuming from a checkpoint, as a night is a long time for a rented machine. Then we rent one RTX 4090 and train for nearly eight hours. In the morning, we measure the model on HellaSwag, a benchmark of common sense that GPT-2 small is often compared on, and in bits per character, against episode twelve's model, which learned from stories. All right, let's get started with where the web pages come from.

Web pages

2:31 FineWeb comes from The FineWeb Datasets, a paper from 2024 by Guilherme Penedo and others at Hugging Face. Its abstract starts from how much a language model depends on the quality and the size of its pretraining data, and points out that the datasets behind open models like Llama 3 aren't public. FineWeb is fifteen trillion tokens of web pages, from ninety-six snapshots of Common Crawl, a crawl of the web that anyone can download. FineWeb-Edu is the part of it judged educational, one point three trillion tokens. Section four of the paper says how the pages were judged. Llama 3, with seventy billion parameters, scored four hundred and sixty thousand pages from zero to five for their educational value, at the level of grade school and middle school. A small model, a linear layer on top of an embedding model, learned to give Llama's scores, and it scored all of FineWeb, which took six thousand hours of H100 GPUs. The pages that score three or more make up FineWeb-Edu. Figure eleven shows what that buys: models trained on it reach a given score on MMLU, a benchmark of school and university questions, with far fewer tokens than with the other open datasets.

3:42 Data dot py gets a dash dash web option. It imports shutil, for copying a download to disk as it arrives, and thread pool executor, for downloading several files at once. FineWeb is where the files are on Hugging Face: FineWeb-Edu's sample of ten billion GPT-2 tokens, which comes as fourteen files of web pages, about two gigabytes each. Web BPE size is thirty-two thousand seven hundred and sixty-eight tokens: the two hundred and fifty-six bytes, thirty-two thousand five hundred and ten merges, padding and end of text, which still fits in the sixteen bits our token files store each id in. The stories had eight thousand one hundred and ninety-two, and the web has far more words to cover. The merges are learned from the first two hundred and fifty-six megabytes of the first file, and the held-out pages are the first ten thousand of the last file, which training doesn't read. Load tokenizer takes the name of the merges file, the stories' merges by default, and so does start worker, which loads the tokenizer in each of the pool's processes. Download shard fetches one of the files, unless it's there already. A file is about two gigabytes, so it's streamed to disk in pieces of sixteen megabytes, under a name ending in dot part, and renamed once it's whole, so a download that stops halfway doesn't pass for a file. Web pieces reads a file's pages, as pieces of about thirty-two megabytes of text, in bytes, each page followed by end of text. The files are Parquet, a format that stores a table by column, and pyarrow reads the text column, a thousand and twenty-four pages at a time. Each page is added to the piece with end of text after it, and when the piece passes thirty-two megabytes, or the pages we asked for run out, it's encoded as UTF-8 and handed out. Build web starts with its comment: pages from the web that a classifier, trained on a big model's ratings, scored as educational, with merges of their own. Four threads download the files we train on, and the last file. If there are no web merges yet, it collects the first file's pieces, with end of text replaced by a space, until they pass two hundred and fifty-six megabytes, and our C++ trainer from episode eleven learns the merges from them, which are written to web merges dot txt. The held-out pages are written to valid web dot txt, and encoded with the encode function we already have. Then the training files are encoded in pieces, a piece per core, as for all of TinyStories in episode twelve. The comment explains max tasks per child: a process gets a fresh tokenizer after four pieces, as the encoder remembers each chunk it has seen, and the web has a lot of them, so a process that lived for the whole run would keep growing. Measured counts the bytes of each piece as it goes by, and the ids of each file are appended to train web dot bin, with a line per file, and the bytes per token at the end. In main, dash dash web takes how many files to train on. Remote dot sh, from episode twelve, gets the rented machine ready. Its usage gets a second line: with dash dash web and a number after the host, it makes the web's data instead of TinyStories. Shift drops the host from the arguments, and data is the rest of them, or dash dash full when there are none. The web merges are copied when there are some, so the rented machine uses the merges learned on the laptop. The commands for the machine are in double quotes now, so data is filled in before they're sent, and pip installs pyarrow as well.

7:08 On the laptop, data dot py dash dash web one makes the web's data from the first file, which is plenty to try it on. I'd downloaded the two files before recording, so it goes straight to the merges: about thirty-two thousand five hundred of them, learned from two hundred and fifty-seven megabytes of pages in about thirty-three seconds. The ten thousand held-out pages become about ten million tokens, at about four point six characters per token, and the first file becomes about seven hundred and fifty million tokens, in about twenty-five seconds on the laptop's twelve cores.

7:40 To see what merges of their own are worth, this line of Python encodes the first two million characters of the held-out pages with each set of merges. With the stories' merges, a token holds about three point two three characters, as words that children's stories don't use, like photosynthesis, fall apart into small pieces. With the web's merges, a token holds about four point six seven, so a page takes about thirty-one percent fewer tokens, and a window of a thousand and twenty-four tokens holds about forty-five percent more text.

A night's budget

8:08 How many tokens does a model of our size need? Training Compute-Optimal Large Language Models, from 2022, by Jordan Hoffmann and others at DeepMind, trained over four hundred models, from seventy million to sixteen billion parameters. Its abstract finds that for a given amount of compute, the model and its data should grow together: each time the parameters double, the tokens should double too. To test it, they trained Chinchilla, seventy billion parameters on one point four trillion tokens, about twenty tokens per parameter, and it beat models four times its size. Figure three shows how they found it: for each amount of compute, a curve of the loss against the model's size, with a valley at the size that gets the most from that compute. Twenty tokens per parameter became the rule of thumb for a model trained to the best loss its compute can buy.

8:57 Our model, in GPT-2 small's shape, with the web's vocabulary of thirty-two thousand seven hundred and sixty-eight tokens, has about a hundred and ten million parameters, so twenty tokens per parameter is about two point two billion tokens. In episode twelve, one RTX 4090 trained a model of ninety-one million parameters at about one hundred and forty thousand tokens a second. Ours is a bit bigger, with twice the context, but at about that speed, eight hours is about three point nine billion tokens, more than Chinchilla's twenty per parameter, and five of FineWeb-Edu's files hold about that many. The cheapest RTX 4090 I could rent cost about forty-six cents an hour, so a night costs about three dollars seventy, and I had four dollars and eleven cents left on Vast.ai from the earlier episodes. One night, then, on one GPU, and one pass over five files.

Overnight

9:49 Pretrain DDP dot py, from episode twelve, gets three options. Web trains on FineWeb-Edu's tokens, with the web's merges. Accum is the number of batches whose gradients add up before each step, and resume carries on from the run's last checkpoint. The tokenizer loads the web's merges with dash dash web, and the stories' otherwise. The run's name and folder move up, as resuming needs them before the model is wrapped: pretrain web for a run on the web, pretrain DDP otherwise, with the modern block and the key and value heads after it, as before. Its comment says what a resumed run carries on from: the weights, the optimiser's state, the step and the clock it saved. Resume loads resume dot pt from the folder, with torch load, and the weights come from the model's checkpoint, into raw, before DDP and compiling wrap it. With dash dash web, the training tokens are train web dot bin, and the held-out ones are valid web dot bin, as the web has pages of its own. Prompt is what the model writes from at each evaluation: photosynthesis is, for the web, as a story's once upon a time doesn't say much about a model of web pages. Once the optimiser is made, a resumed run loads its state, AdamW's two running averages for each weight. Evaluate writes from the prompt now, and after saving the model, it saves resume dot pt as well. The comment says what's in it: what it takes to carry on, the optimiser's averages, the step, and the seconds trained so far. AdamW keeps two numbers for each weight, so it's about twice the size of the model. A resumed run's clock starts where the saved one stopped, by moving the start back by the seconds it had trained. So the share of the budget that has gone by, and the learning rate's schedule with it, pick up where they were. The step comes from resume dot pt too, and a resumed run skips the evaluation at its first step, which was the saved one's last. In the loop, zero grad comes first now, and then accum batches, each with its forward and backward pass. Backward adds to the gradients that are there, so after accum batches, they add up to the gradient of one batch accum times as big, once each loss is divided by accum. DDP would average the gradients across the GPUs after each backward, so no sync turns that off for all but the last batch, and they're averaged once per step. Then the clipped gradients and the optimiser's step, as before. The speed counts accum batches per step, on each GPU, and so do the tokens at the end.

12:18 On Vast.ai, search offers asks for one RTX 4090 this time, with a reliability over ninety-nine percent, a fast download, and at least thirty gigabytes of memory, for the data. Dash o d p h sorts them by the price per hour, with eighty gigabytes of disk, and jq prints a line per offer, with how many cores we get. A few minutes before, there were cheaper ones, in Poland at thirty-nine cents an hour, but they'd been rented by the time I recorded this. So we take the cheapest, in Japan, at about zero point four five nine dollars an hour, with sixteen cores.

12:51 Create instance rents it, with PyTorch's own image, eighty gigabytes of disk, and ssh straight to the machine, labelled mvgpt dash web. It answers with the new instance's id, and an API key for the instance, which I've hidden.

13:08 About a minute later, show instances has it running, and I've given its address to ssh under the name vast. On the machine, nvidia smi shows one RTX 4090, idle, with about forty-nine thousand megabytes of memory, twice a standard 4090's twenty-four gigabytes. Some 4090s in data centres have been rebuilt with twice the memory chips, and this is one of them. We don't need the extra memory: the run is going to use about fourteen point five gigabytes at most, which fits in a standard 4090.

13:38 Remote dot sh vast dash dash web five copies the code and both sets of merges, builds the tokenizer, and runs data dot py dash dash web five on the machine. The held-out pages become the ten million tokens they did on the laptop, and the five files become about three billion eight hundred million tokens, at about four point six bytes per token, a file about each minute. Python counts one hundred and twenty-eight cores, the whole machine's, though our share of it is about sixteen, so the pool starts more processes than we have cores, which works, with more switching between them. With the downloads, the setup took about thirteen minutes.

14:16 Then the night's run: four hundred and seventy minutes, compiled, with a context of a thousand and twenty-four tokens, and four batches of twenty-four sequences per step, so each step trains on about ninety-eight thousand tokens. Nohup keeps it going after we disconnect, its output goes to train dot log, and its process id to train dot pid. Tail follows the log for the first five minutes. The model has one hundred and ten million parameters, and at step zero, the held-out loss is about ten point six, close to the log of the vocabulary's size, as an untrained model spreads its guesses over all the tokens, and what it writes after photosynthesis is, is random tokens. After compiling, it trains at about one hundred and forty thousand tokens a second. By step two hundred and fifty, the loss is about five point four, and photosynthesis is a type of to reduce the benefits of health: words that go together, in sentences that don't say much yet.

15:11 In the morning, about a quarter of an hour before the end, tail dash f follows the log again, until the training process exits. The learning rate is down at the end of its cosine, a tenth of where it started, and the speed held at about one hundred and forty thousand tokens a second all night. The samples have changed: photosynthesis is the process of taking carbon from the air and producing oxygen, and turning it into sugar, which is close, though not quite how it goes. In the last one, photosynthesis happens in stars, written with as much confidence as the rest: the model writes like the pages it read, whether or not what it writes is true. At the end, it took thirty-seven thousand steps, about three billion six hundred million tokens, which is zero point nine six of a pass over the five files, or about thirty-three tokens per parameter, in four hundred and seventy point two minutes, and the held-out loss ends at about three point one one.

16:01 Through the night, the held-out loss falls fast in the first hour, to about three point five six, then more and more slowly: about three point two seven at four hours, and three point one one at the end, still going down. The training loss stays next to it, as the model reads almost each page once, so it can't memorise them: a bigger budget would have kept buying lower losses.

16:23 Then the model had to come back to the laptop, four hundred and forty megabytes, and from Japan, scp managed about a quarter of a megabyte a second, which would have taken longer than the credit left. So I split the file into eight parts on the machine and copied them at once, each over its own connection, and joined them back together here. The copy's SHA-256 checksum, its fingerprint, matches the one the machine printed for the original, so it arrived intact. Then I destroyed the instance, and show instances has none left. From renting it to destroying it, about eight point six hours, the night came to about three point nine nine dollars, with zero point one two dollars of credit to spare.

What a night bought

17:01 Back on the laptop, sample dot py writes from three beginnings. From the French Revolution, it writes like a page of history: liberty and equality, and seventeen eighty-nine. It also gets things wrong, just as confidently: the French Revolution wasn't the Revolutionary War, or a war of independence, and France wasn't the first country to adopt a democratic government. Asked how to grow tomatoes at home, it writes a list, as a page on gardening would, though the list goes round in circles: some tomatoes prefer one soil, some another, as a small model's most likely next line is often one like the last. And from that beginning, episode twelve's model, which learned from stories, turns it into a story about Tim and a little girl who plant seeds together. Each of the two models writes like what it read.

HellaSwag

17:49 Samples are hard to compare, so we measure the model on a benchmark. HellaSwag: Can a Machine Really Finish Your Sentence?, from 2019, is by Rowan Zellers and others at the University of Washington and the Allen Institute for AI. Its abstract describes the task: given the beginning of a scene, pick the most likely way it carries on, from four endings. The wrong endings were written by a language model, and kept when they fooled the classifiers of the time, so they're on topic, but they don't make sense. People get over ninety-five percent of them right, while the best models of 2019 got under forty-eight. Figure one shows two of them: one from ActivityNet, captions of videos, and one from WikiHow, how-to articles. A model that wasn't trained for the test can still take it: we score each ending by how likely the model finds it, after the beginning, and its pick is the most likely one. Andrej Karpathy scores his reproduction of GPT-2 small this way, in his video and in llm.c.

18:50 The scoring goes in a new file, hellaswag dot py. It imports argparse, json, os and urllib, then torch and its functional module, and from our own files, load checkpoint, the data folder and pick device. The comment at the top describes the data: HellaSwag's validation set, from its copy on Hugging Face, ten thousand and forty-two beginnings of a scene, each with four endings. People pick the right one about ninety-five percent of the time, and guessing gets twenty-five. The dataset's own GitHub repository has been taken down, so the copy on Hugging Face is the one we read. Results is the folder the scores are saved to. Load examples downloads the Parquet file once, into the data folder, and pyarrow reads three columns from it: the beginning, the four endings, and the label, the number of the right ending, which comes as text, so it's turned into an int. Ending losses scores one example, all four endings at once. The beginning is encoded once, and each ending with a space in front, as it carries on the sentence. Each row is the beginning followed by an ending, and the four rows are padded with zeros to the longest. Scored marks the ending's positions in each row, so that the ending's tokens are the ones that count: how surprised the model is by each ending, given the beginning. The forward pass gives logits for the four rows at once. Each position predicts the next token, so the logits up to the last line up with the tokens from the second on, and cross entropy with reduction none gives a loss for each token, and multiplying by scored keeps the ending's. The sum favours short endings, which have fewer tokens to be surprised by, so the score that counts is the average over each ending's tokens, and it returns both. Main takes a checkpoint, or gpt2, for OpenAI's GPT-2 small, which transformers downloads from Hugging Face with its tokenizer. Forward and encode wrap each kind of model, so the scoring code doesn't care which it has. For each example, the pick is the ending with the lowest loss, by the average and by the total, and each thousand examples, it prints the accuracy so far. At the end, it prints both accuracies, and saves them with the model's name and size.

21:00 First, GPT-2 small, which transformers downloads from Hugging Face, with its one hundred and twenty-four million parameters, trained by OpenAI in 2019 on about forty gigabytes of web pages. The first three thousand two hundred and forty-three examples come from ActivityNet, and it gets about a third of them right. The WikiHow ones, which come after, are harder for it, and it ends at about twenty-nine point five five percent, against twenty-five for guessing. That's the score usually quoted for GPT-2 small, so our way of scoring matches the usual one. Episode twelve's model, which has read stories and little else, gets about twenty-seven point three five percent, a little above guessing. A scene of someone washing a dog isn't far from a children's story, while a how-to article on taxes is.

21:47 And now the model we trained through the night. On the first examples, from ActivityNet, it gets about a third right, as GPT-2 small did, and on the WikiHow ones it falls off in a similar way. It ends at about twenty-nine point four five percent, two thousand nine hundred and fifty-seven right, against GPT-2 small's two thousand nine hundred and sixty-seven, so ten examples apart, out of ten thousand and forty-two. On a test this size, a score moves by about half a percent from chance alone, so a gap of a tenth of a percent says the two do about as well.

22:20 Side by side, guessing gets twenty-five percent, and episode twelve's model, which read stories, gets about twenty-seven point four. One night on educational web pages brought a model of about a hundred and ten million parameters level with GPT-2 small, which has about fourteen million more, and which OpenAI trained on WebText, forty gigabytes of pages linked from Reddit. GPT-2 small was the state of the art in 2019, so this is far from today's models, but it's a fair measure of how far four dollars and a night go now.

Bits per character

22:52 Bits per character, from episode eleven, compares models across tokenizers, as it measures the loss per character of text, whatever the tokens are. Bits dot py gets a dash dash web option, which reads the first two million characters of the held-out web pages, instead of the stories, so each model can be measured on the other's text.

23:14 Bits dot py measures both models on the held-out stories first. Episode twelve's model, on its home ground, needs about zero point three eight six bits per character. The web model, which wasn't trained on a single story collection, needs about zero point nine four six, more than twice as much: stories are a narrow kind of text, and a model that learned them alone predicts them very well. With dash dash web, on the held-out web pages, the order flips, by far more. The web model needs about zero point nine five six bits per character, and the stories model about two point four zero, about two point five times as much. The web model's two numbers are close, as children's stories are text that the web covers too, while the stories model hasn't met most of what the pages are about.

23:59 Side by side, each model does best on the text it learned from. The stories model is a specialist, very good at stories and lost on the web, and the web model is a generalist, about as good on stories as on web pages.

24:13 Last, the docs get a page on pretraining on the web, with FineWeb-Edu and its merges, the night's budget against Chinchilla's rule, the new options, and the run's results on HellaSwag and in bits per character, and it goes at the end of the navigation.

Wrap-up

24:29 Now, let's go over what's real here, and what's scaled down. The data is real: FineWeb-Edu is what far bigger models are trained on, and we read it as it comes from Hugging Face. So is the recipe: GPT-2 small's shape, AdamW and a cosine schedule, the way models of this size are trained. The scale of the run isn't, not by a long way. We read about three billion six hundred million tokens, while FineWeb has fifteen trillion, and Llama 3 was pretrained on over fifteen trillion, about four thousand times as many. And we had one GPU for a night, where models like that train on thousands of GPUs for weeks. So our model writes fluent pages, and scores as well as GPT-2 small on HellaSwag, but it knows few facts, and it doesn't follow instructions: it's a pretrained model, which carries on whatever text it's given.

25:24 So, today we pretrained our model on pages from the web. FineWeb-Edu's pages, which a classifier scored as educational, got BPE merges of their own, which fit about forty-five percent more text in a window than the stories' merges. Chinchilla's twenty tokens per parameter put our hundred and ten million parameters at about two point two billion tokens, and a night on one GPU read about three billion six hundred million. Pretraining got the web's data, gradient accumulation, for steps of about ninety-eight thousand tokens, and resuming from a checkpoint. One RTX 4090 in Japan trained for four hundred and seventy minutes, at about one hundred and forty thousand tokens a second, for about three point nine nine dollars, and the held-out loss went from about ten point six to three point one one. On HellaSwag, the model gets about twenty-nine point four five percent, ten examples short of GPT-2 small, and in bits per character, it's about as good on stories as on web pages, while episode twelve's model is good at stories alone.

26:23 If you've pretrained a model of your own on a budget, let me know in the comments what it read, and how far your dollars got you. And if you'd like to follow along, subscribe to the channel. I hope you find this useful, and thanks for watching.