BPE in C++: a byte-level tokenizer for a GPT

Intro

0:00 Hello and welcome back to episode eleven, where we're going to write a byte-pair encoding tokenizer, the kind GPT-2 uses, in C++, and pretrain our model with it. Let's begin with a quick look at where we're going to be by the end of the episode.

0:15 We give last time's model the start of a story with a name it hasn't seen, Zorblax, and the name is gone, as it isn't one of the ten thousand words its tokenizer knows. The model we pretrain today gets the name as five tokens, and it stays in the story, as its tokenizer can encode any text, down to single bytes.

0:34 On the website of the series, you can find more information about the project. The code is open source on GitHub, tagged at each step of each episode, so that you can follow along, and the docs get a page on the BPE tokenizer today, with its speed against Python and how the two tokenizers compare. And each episode has a page here as well, with its chapters and its code.

Our tokenizer, from episode two

0:57 Now, before we get to today's work, let's quickly go over what we did last time. We replaced the model's table of learned positions with rotary position embeddings, which turn each head's queries and keys by an angle that grows with the position. Attention then sees how far apart two tokens are, and the KV cache can slide past the context window, a token at a time. One part of the model hasn't changed since episode two, and that's its tokenizer. It splits text into words and runs of punctuation, and keeps the ten thousand most frequent tokens of the stories. Any other word becomes unknown, and spaces and new lines are dropped, then put back by rules when we decode, which is why, in episode five, Hugging Face's decoder put spaces around the punctuation.

1:40 Today, we're going to replace it with byte-pair encoding, or BPE, the tokenizer of GPT-2 and of most models since. We write it in C++, which is the language I'm more familiar with, and build it into a module that Python can import, with a version in plain Python next to it, which the tests check the C++ against. Then the model gets a tokenizer built on that module, and we tokenize the stories with it and pretrain for ten minutes. Last, we measure how much faster the C++ is, and compare the two tokenizers' models in bits per character, as their losses per token can't be compared. All right, let's get started with what byte-pair encoding does, in the paper it comes from.

Byte-pair encoding

2:20 Byte-pair encoding for language comes from this paper, Neural Machine Translation of Rare Words with Subword Units, by Rico Sennrich, Barry Haddow and Alexandra Birch, at the University of Edinburgh, published in 2016. Its abstract starts from our problem: a translation model has a fixed vocabulary, but the texts it translates have words outside it. So the authors encode rare and unknown words as sequences of smaller units. Byte-pair encoding began as a way to compress data, from Philip Gage in 1994, which replaces the most frequent pair of bytes with a byte the data doesn't use. The paper merges characters instead: each word starts as its characters, and the most frequent pair of neighbouring symbols becomes a new symbol, again and again, for as many merges as we ask for. Algorithm one is a minimal version in Python, which counts the pairs, picks the most frequent, and merges it in each word of the vocabulary. Under it, the paper notes that pairs don't cross word boundaries, and that in practice, they index all the pairs and update them as they merge, instead of counting again, which is what our C++ is going to do. And figure one shows the merges learned from four words: r with the end of a word first, then l and o, lo and w, and e with r at the end of a word.

3:36 GPT-2 made it byte-level: the symbols it starts from are the two hundred and fifty-six possible bytes, not the characters. So any text can be encoded, in any language and with emoji, and no token is unknown, as a character outside plain English is a few bytes in UTF-8, which merges can join back up. These are real merges, from the tokenizer we're going to train on the stories. A word such as upon, with the space in front of it, starts as five bytes. O and n are the forty-eighth merge, a space and u the sixty-eighth, then space u and p, and the hundred and ninety-fourth merge makes space upon a single token. A name the stories don't have, such as Zorblax, is still encoded, as five tokens, where the word-level tokenizer gives it a single unknown token.

In Python first

4:24 We start in plain Python, with bpe reference dot py, as it's short, it follows the paper, and it's what the C++ is going to be checked against. It imports re, for regular expressions, and Counter, a dictionary that counts. Chunk is the pattern that splits the text before any merging. It works on bytes, as the b in front of the string says, and it matches a run of letters, digits and the bytes of characters outside plain English, or a run of punctuation, each with an optional space in front, or else a run of whitespace. Those are the chunks that merges don't cross, so space upon can become a single token, while time and the comma after it stay apart, and pretokenize returns the chunks it finds. Merge takes a list of ids, a pair, and the id of the merged token. It goes left to right, writing the merged id wherever the pair appears and skipping both of its ids, or copying the id across. Train learns the merges, and Counter counts each distinct chunk, so a chunk that appears a hundred thousand times is worked on once, along with its count. Each chunk becomes a list of its bytes, as going through bytes in Python gives integers from zero to two hundred and fifty-five. Then, for each merge we ask for, we count the pairs of neighbouring ids, where zip of the ids and the ids shifted by one gives the pairs, weighted by how often their chunk appears, and if no pair is left, we stop early. The pair we merge is the most frequent, and of the most frequent, the smallest, so when two pairs tie, the reference and the C++ both pick the smaller one. Min with that ordering does both, as minus the count puts the most frequent first. The new token's id comes after the two hundred and fifty-six bytes and the merges before it, and we apply the merge to each chunk. Counting all the pairs again for each merge is simple, and slow, as each of thousands of merges goes over all of the chunks. Encode applies the merges to new text, where ranks gives each pair the order it was learned in. For each chunk, we start from its bytes, and while there's more than one id, we find the neighbouring pair with the lowest rank, where a pair that isn't a merge gets a rank past all of them. If even that pair isn't a merge, the chunk is done, and otherwise we merge it and look again. Applying the merges in the order they were learned gives the tokens that training would have made of that text.

In C++

6:44 Now the C++, in bpe slash bpe dot cpp, which does this work faster, by keeping the counts up to date instead of counting again. Each include line pastes in the declarations of what we use from a header. Pybind eleven is a library that makes C++ functions and classes callable from Python, and the file's first line, over it, says what the file is: byte-level BPE, training merges on a text and encoding text with them, exposed to Python with pybind eleven. Its stl header converts the standard library's containers, such as vectors and pairs, to Python's lists and tuples, and back. From the standard library, c std int has integers of a fixed size, such as an unsigned sixty-four-bit integer, and queue has the priority queue, which is going to keep the pairs' counts. String holds bytes, and string view looks at part of a string without copying it. Unordered map is C++'s hash map, like a Python dictionary, and unordered set its set. Utility has pair, and vector is the standard library's growable array, like a Python list. Namespace py equals pybind eleven gives the library a short name. The namespace with no name that follows keeps everything inside it to this file, so these helpers can't clash with another file's names, as the module at the bottom is all that Python needs to see. Using Pair names a type, a pair of two ints, for a pair of neighbouring token ids. Is word says whether a byte belongs inside a word. An unsigned char is a byte, from zero to two hundred and fifty-five, and a function declared as bool returns true or false. It's true for a letter or a digit, or for a byte of a hundred and twenty-eight and up. Those are the bytes of characters that take several bytes in UTF-8, which stay inside words, while is space checks for the six whitespace characters. Pretokenize does by hand what the reference's regular expression does: it splits text into chunks that merges don't cross, a run of word bytes or of punctuation, each with an optional space in front, or a run of whitespace. It takes a string view of the text, and returns a vector of string views, one per chunk, so the chunks point into the text without copying it. Size t is the type C++ uses for sizes and positions, and i is where we are in the text. For each chunk, we note where it starts and read its first byte, where static cast converts it to an unsigned char, as a plain char can be negative. A space followed by something that isn't whitespace is a leading space, so it joins the chunk, and we move on to the next byte. If the byte is a word byte, the inner while loop moves i on to the end of the run of word bytes. Otherwise, it's a run of punctuation, which is neither a word byte nor whitespace, or else a run of whitespace. Substr gives the view from the start of the chunk to here, and push back adds it to the chunks, which we return at the end. Key packs a pair into one unsigned sixty-four-bit number, the first id shifted up by thirty-two bits and the second in the lower thirty-two, as a single number makes a quick key for a hash map. Merge is the reference's merge, done in place, as it replaces each pair in ids, left to right, with the new token. The ampersand after the type makes ids a reference, so the function changes the caller's vector instead of a copy. Out is where the next id goes, and where the pair appears, we write the new token and skip both ids. As the list can't grow, we can write over it as we read it, then resize it to what's left. Train learns the merges, where each time, the most frequent pair of neighbouring tokens becomes a new token. It takes the text as a const reference, so it's neither copied nor changed, and returns the merges as a vector of pairs. Counts maps each distinct chunk to how often it occurs, so each distinct chunk is worked on once, as its bytes, with its count. Its keys are string views into the text, so counting a chunk copies none of it. For each chunk of the text, plus plus adds one to its count, which starts at zero. Then words holds each distinct chunk as a vector of ints, and freq its count, side by side. Auto lets the compiler work out a type, and the square brackets unpack each entry of the map into its chunk and its count. Emplace back builds the word in place from the chunk's bytes, and static cast turns each one into a number from zero to two hundred and fifty-five, like the reference's bytes. Next, pair counts maps each pair's key to how often it occurs, and where maps it to the words it occurs in, as a set, so a merge can go straight to those words. Count is a lambda, a function written inline, and the ampersand in its square brackets lets it use the variables around it, by reference. It goes over a word's pairs, and adds the word's frequency to each pair's count, or takes it away with a sign of minus one, and when it adds, it records the word under the pair. And we count each word's pairs once, to start. The heap is where the speed comes from, and order is its comparison. It puts the most frequent pair on top, and between pairs with equal counts, the smaller pair, which is the reference's tie-break. The comment warns that entries go stale: a merge takes occurrences away from other pairs, so when an entry comes out of the heap, we check it against pair counts. Decltype of order hands the lambda's type to the priority queue, and we push each pair with its count. Merges starts empty, and the loop runs until it has as many as we asked for, or the heap is empty. It takes the top entry and pops it, and if the pair is gone, or its count is down to zero, we skip it. If its count has changed since it was pushed, the entry is stale, and it goes back in with the count it has now, so we try the next one. Otherwise, it's the most frequent pair, and it becomes the next merge, with the id after the bytes and the merges so far. The words that hold the pair are the ones that change, so for each of them, we take its pairs out of the counts, merge it, put its new pairs back in, and push those pairs, so the heap has their new counts. Last, we erase the merged pair's count, as all of its occurrences have been merged. Each merge then costs time for the words that hold the pair, and not for the whole text. A first version of this dropped the stale entries instead of pushing them back, and the tests caught it, as its merges came out different from Python's. The encoder class encodes text with the merges, and its constructor is marked explicit, as a constructor with a single argument would otherwise also turn a vector of merges into an encoder wherever one is expected. It fills ranks, from each merge's key to the order it was learned in, where the underscore at the end of the name marks a member of the class. Encode goes chunk by chunk, and looks each one up in the cache first. If it's not there, emplace encodes it and keeps its ids, and either way, insert appends them to the result. Encode chunk is private, as it's for the class's own use, and the const after it says it doesn't change the encoder. It applies the merges in the order they were learned, the pair with the lowest rank each time, until none is left. It starts from the chunk's bytes, as numbers from zero to two hundred and fifty-five. While there's more than one id, best starts at minus one, for no merge found yet. For each pair of neighbours, find looks up its rank, and if it's a merge, and learned earlier than the best so far, it becomes the best. If no pair is a merge, the chunk is done, and otherwise we merge the best pair into the id two hundred and fifty-six plus its rank, and look again. The class keeps ranks and the cache, which maps a chunk to its ids. Chunks repeat a lot, such as once, space upon and space a, so most of a story's chunks are found there. And the closing brace ends the namespace with no name. Last comes the module, where PYBIND11 MODULE is a macro, code that pybind eleven expands into the function Python calls when it imports bpe, and m is the module, with a doc string. M dot def adds a function, and pretokenize takes py bytes, a Python bytes object, which converts to a string by copying the bytes. It returns the chunks as a list of Python bytes, for the tests. Train takes its text as bytes too, and gil scoped release lets go of Python's global interpreter lock while it trains. The C++ doesn't touch Python's objects, so other Python threads can carry on in the meantime. And py class makes the encoder a Python class, whose constructor takes the merges as a list of pairs, with an encode method, so that Python can write bpe dot Encoder of the merges, dot encode.

Built, and tested against Python

15:22 To build it, we write setup dot py, which is how Python packages are built, here with a single extension in C++. Pybind eleven's setup helpers give us the extension and its build step, and python setup dot py build ext dash dash in place is the command that builds it, putting the compiled module next to our code, while setup comes from setuptools. The extension is named bpe, built from bpe dot cpp, with C++ seventeen, the version that added string view, and dash O three, the compiler's highest level of optimisation, and cmdclass hands the build step to pybind eleven's. Pybind eleven and pytest join the requirements, and git ignores the build folder and the compiled module, which setup dot py can make again. Then the tests, in tests slash test bpe dot py, for pytest, which runs each function whose name starts with test, and reports the ones that fail. They import pytest, the C++ module, bpe, and the reference. Data is our training stories, found from the test's own folder, and sample is their first five hundred thousand bytes, or a made-up story repeated, if the data isn't there. The first test checks that both split the sample into identical chunks, where bytes of each chunk turns pybind eleven's bytes into Python's, so the two lists compare. The second checks that both learn two hundred merges, in an identical order, where tuple of m turns each of the C++'s pairs, which arrive as lists, into a tuple like the reference's. A heap that handles its stale entries wrong would fail here. The third learns merges with the reference, and checks that the C++ encodes the last fifty thousand bytes into the ids the reference gives. Parametrize, from pytest, runs the next test once for each of three texts: a sentence from a story, one with accents, Japanese and an emoji, and one with double spaces, blank lines and a tab, which the word-level tokenizer would lose. Each of them is a case of its own in pytest's report, so a failure says which text it was. The round trip learns merges with the C++, encodes the text as UTF-8, and rebuilds each token's bytes, the two hundred and fifty-six single bytes, then each merge's two parts joined. Joining the ids' bytes and decoding them has to give the text back as it was.

17:38 Let's build it, with dash q to keep setuptools quiet, and the compiler leaves a module named bpe, for Python three point twelve on darwin, which is what Python calls macOS. The one warning is the linker's, about a folder that Python's own build used, which isn't on this machine. Then pytest, where dash v lists each test, and all six pass: the chunks, the two hundred merges and the ids match the reference's, and the three texts come back as they were, the emoji along with them.

Where to go deeper

18:08 To go deeper, Andrej Karpathy's Let's build the GPT Tokenizer writes byte-pair encoding in Python from scratch, over about two hours, and goes through the tokenizers of GPT-2 and GPT-4, along with the oddities of language models that come from how they cut up text.

18:23 And his nanochat, a small ChatGPT, also leaves its tokenizer's work to compiled code, and calls it from Python. Its tokenizer's docstring says it trains with rustbpe, a BPE trainer he wrote in Rust, and encodes with tiktoken, OpenAI's tokenizer library. Its split pattern, further down, is GPT-4's regular expression for chunks, with numbers cut into groups of up to two digits instead of three, as its comment explains.

A tokenizer for the model

18:50 Now the model needs a tokenizer built on the module, next to the word-level one, in tokenizer dot py. BPE specials split finds our padding and end-of-text tokens in a text, as those two are whole tokens of their own, which don't go through the merges. The BPE tokenizer class imports bpe inside its constructor, so the rest of tokenizer dot py still works on a machine that hasn't built the module, and the command that builds it goes next to the import. It keeps the merges as tuples, and makes an encoder from them. Token bytes holds each token's bytes: the two hundred and fifty-six single bytes, then, for each merge, its pair's bytes joined, so a merged token's bytes come from the two tokens it was made of. Padding and end of text get the two ids after them, so the vocabulary is the bytes, the merges and those two. Unknown is None, and over it we write why: any text is bytes, and each byte is a token, so there's no unknown token. Id to token gives each token as text, for printing, where a token that ends in the middle of a character gets a replacement mark. Len gives the size of the vocabulary, as the word-level tokenizer's does. Encode starts with an empty list of ids, and splits the text on padding and end of text, where the group in the pattern keeps those two among the pieces. Each of those two gets its id, and any other piece that isn't empty is encoded as UTF-8 bytes by the C++, which hands back its ids. Decode goes through the ids, and each token with bytes adds them to a list. Padding and end of text add their text when we ask for them, which we don't by default, so a story prints without them. At the end, decode joins all the bytes first and then decodes them as UTF-8, as a token can end in the middle of a character. A checkpoint now has to record which tokenizer its model was trained with. Save checkpoint keeps the model's weights and its config, then a BPE tokenizer's merges, or a word-level one's vocabulary. And load checkpoint makes a BPE tokenizer when the checkpoint has merges, and a word-level one otherwise, so the models we trained before load as they did.

The stories, in BPE

20:57 In data dot py, we import sys, for a flag, and time, to time the tokenizers, along with the BPE tokenizer. BPE size is eight thousand one hundred and ninety-two tokens: the two hundred and fifty-six bytes, seven thousand nine hundred and thirty-four merges, and padding and end of text, a little fewer than our ten thousand words. Encode now reads the text first, and times the encoding on its own, from start to the moment the ids come back, as reading the file isn't the tokenizer's work. A tokenizer with no unknown token counts none, and the line it prints gains the characters per token, which is how much text a token holds on average, and the seconds the encoding took. Load tokenizer takes a bpe flag, and then reads the merges from merges dot txt, a pair of ids per line. Build BPE imports the module inside the function, like the tokenizer does, and learns the merges from the stories and the instructions, as the word-level vocabulary was, so fine-tuning's words are in it too. It reads each file into one text, and end of text becomes a space, as it's a token of its own, and stays out of the merges. We time the training, print how many merges it learned from how many megabytes, and write them to merges dot txt, a pair per line. Then the tokenizer encodes the training and validation stories, to train dash bpe dot bin and valid dash bpe dot bin, next to the word-level files. In main, dash dash bpe does all of that, then prints a sentence, Once upon a time, a cat said hello, in quotes, along with its tokens, their ids and the text we decode from them, and returns before the word-level vocabulary. So the BPE files come from a flag of their own, and data dot py without it does what it did before, which keeps the word-level files for the models we've trained on them. In pretrain dot py, main takes a flag for BPE as well, and the run's folder gets its name from both flags. Each kind of model gets a run folder of its own, pretrain, pretrain dash rope or pretrain dash rope dash bpe, so they don't overwrite each other. The tokenizer and the token files follow the flag, and the vocabulary's size comes from the tokenizer, as it did, so the model gets a row in its embedding table for each of the eight thousand one hundred and ninety-two tokens. And the command line passes dash dash bpe through to main, next to dash dash rope.

23:12 Python data dot py dash dash bpe learns the merges from about one hundred and fifty-eight megabytes of stories and instructions, in about one point five seconds. Then it encodes the training stories as about twenty-one million tokens, at about four point zero eight characters per token, with none of them unknown, in about one point eight seconds. The sentence comes out as eleven tokens, as many as the word-level tokenizer gave it in episode two. Here, though, each word carries the space in front of it, so decoding joins the bytes, and the sentence comes back as it was, with no rules to put the spaces in.

Pretraining on it

23:47 Now we pretrain on them for ten minutes, with rotary embeddings, like last time's model, and the BPE tokens. The model has about thirteen million eight hundred thousand parameters, fewer than last time's, as its embedding table has fewer rows. It gets through about one thousand two hundred and thirty-four steps, at about twenty-seven thousand tokens a second, and its held-out loss goes from about nine point one to about two point one two. That loss is per token, and a BPE token is a different unit from a word-level one, so we can't put it next to last time's loss to say which model is better. We're going to compare them per character instead, once we've measured the speed.

How much faster

24:26 To measure the speed, bpe bench dot py imports os and time, the C++ module and the reference, and the data folder. It learns five hundred merges from the first tenth of a megabyte, and from the first megabyte, of the training stories, as the Python reference would take far too long on all of them. Timed calls a function, and returns its result along with the seconds it took, from perf counter, Python's most precise clock. Main reads the first megabyte, as bytes, and prints a header. For each size, it trains with the reference and with the C++, and checks that both learned identical merges before it prints anything, or the times wouldn't mean much. Then it prints the two times, and how many times faster the C++ is. Encoding is measured like the training, on the text we trained on, with the reference's merges in both, checked to give identical ids before the times are printed. And main runs when we call bpe bench dot py from the command line.

25:27 On a megabyte of stories, Python takes about two point three seconds to learn five hundred merges, and the C++ about twenty milliseconds, about one hundred and twenty times faster. Encoding that megabyte takes Python about zero point four seven seconds, and the C++ about eleven milliseconds. Learning merges on ten times the text takes neither of them ten times as long, as both work on each distinct chunk once, and a tenth of a megabyte already has many of the chunks a megabyte has.

25:54 Side by side, the C++ learns merges about one hundred and forty times faster on a tenth of a megabyte and about one hundred and twenty times faster on a megabyte. For encoding, the gap grows with the text, from about thirty-one to about forty-two times, as the C++ keeps a cache of the chunks it has encoded, which a longer text finds more often, while the reference encodes each chunk again. On all the stories and instructions, the C++ learned its seven thousand nine hundred and thirty-four merges in about one point five seconds.

Bits per character

26:25 Our loss is the average, over the tokens, of minus the log of the probability the model gave the right one, in nats, as it uses the natural log. Two tokenizers cut a text into different numbers of tokens, so a model whose tokens hold more text makes fewer predictions, each of them harder, and its loss per token can't be set against the other's. So we add the loss up over a text, instead of averaging it, divide by the log of two, which turns nats into bits, and divide by the number of characters, which doesn't depend on the tokenizer. The result is how many bits the model needs, on average, for each character of the held-out stories, and the lower it is, the better the model predicts the text.

27:05 Bits dot py measures that, and imports argparse, math, for the log, os, torch and its functional module, our checkpoint loader, the data folder, pick device, and end of text. Characters is how much of the validation stories it reads, two million. Bits per character runs without gradients, and encodes the text with the model's own tokenizer. It cuts the tokens into back-to-back windows of the model's context, where each token is predicted from the ones before it in its window. N is how many whole windows fit, and x and y are the windows and the windows shifted by one, as two tables of n rows. Nats starts at zero, and we go through the windows sixteen at a time, run the model on them, and add their loss to nats, where cross entropy with reduction sum adds the loss up instead of averaging it. The windows leave out the last few tokens, so the characters are scaled to the share of the tokens they cover, and end of text counts as one character. Unknown is the share of unknown tokens, for a tokenizer that has them. Stories splits the text on end of text, and we keep count of the stories that come back from encoding and decoding as they were, spaces and new lines along with them. Round trip is their share of all the stories. It returns the bits per character, the nats added up, divided by the log of two and by the characters, with the tokens, the share unknown, the characters per token and the round trip. In main, argparse describes the script as bits per character on held-out stories, for models with any tokenizer. It takes one or more checkpoints, and a device, which pick device chooses by default, the Mac's GPU on my laptop. Main reads the first two million characters of the held-out stories, and prints a header, with a column for each number. Then, for each checkpoint, it loads the model with its tokenizer, measures it, and prints a row, named after the checkpoint's folder. And main runs when we call bits dot py from the command line.

29:03 We measure three models, of ten minutes of pretraining each: episode two's, with words and learned positions, last time's, with words and rotary embeddings, and today's, with BPE and rotary embeddings. The word-level tokens hold about four point one six characters each, and BPE's about four point zero one, with no unknown tokens. About zero point six percent of the held-out stories come back as they were from the word-level tokenizer, mostly because of their new lines, and all of them from BPE. In bits per character, the word-level model with rotary embeddings is lowest, at about zero point seven four four, and BPE's is about zero point seven six three, about two point six percent higher.

29:41 The word-level model has an easier job in bits per character. Spaces and new lines count as characters, but it doesn't predict them, as its rules put them back. And it predicts a word outside its vocabulary as a single unknown token. The BPE model predicts each byte, along with the spelling of each name it hasn't seen. So with a smaller vocabulary and fewer parameters, BPE ends a few percent behind the word-level model with rotary embeddings, while it gives back each story as it was, and takes any text we give it.

30:11 Last, the docs get a page on the BPE tokenizer, with what byte-pair encoding does, the C++ module and its Python reference, the tests, the speed against Python, and the three models in bits per character, with what the word-level model doesn't have to predict. It goes at the end of the navigation, after rotary position embeddings.

Wrap-up

30:32 So, today we replaced the word-level tokenizer with byte-pair encoding, which starts from the two hundred and fifty-six bytes, and learns merges of the most frequent pairs. We wrote it in C++, with a heap of the pairs' counts that each merge updates, and tested it against a reference in plain Python, which learns identical merges and gives identical ids. On a megabyte of stories, the C++ learned its merges about one hundred and twenty times faster than Python. The model got a tokenizer on top of it, with no unknown token, and ten minutes of pretraining on the stories in BPE. In bits per character, it ended a few percent behind the word-level model, which doesn't predict spaces and new lines, while BPE gives back each held-out story as it was, along with names that weren't in its training.

31:18 Let me know in the comments how the C++ compares with the Python on your machine, and which parts of tokenization you'd like us to go deeper on. And if you'd like to follow along, subscribe to the channel. I hope you find this useful, and thanks for watching.