Episodes
Build a GPT from scratch in PyTorch
A small GPT written from scratch in PyTorch, from its parameters to how a prompt flows through it to the next token.
Pretraining a GPT from scratch in PyTorch
From random tokens to short stories in a first training run on a laptop, with a tokenizer, the next-token objective, a training loop and evaluation written from scratch.
SFT: fine-tuning a GPT to follow instructions
Our pretrained model learns to write the story an instruction asks for, with a loss on the story alone, and we measure how much better it follows instructions.
KV cache: making a GPT write faster
We add a KV cache to our model, so it runs each new token on its own and not the whole story again, and we check and time it on the CPU and the GPU.
Deploy a GPT: Hugging Face, Gradio, a browser
We save our model as a GPT-2 and push it to the Hugging Face Hub, with a tag per episode, serve it from a Gradio app on a VPS, and run it in the browser with Transformers.js.
Profiling a GPT with PyTorch's profiler
We look at our model as a graph of operations in Model Explorer, profile it with PyTorch's profiler, and measure how decisions in its code change its speed on a CPU and a GPU.
DPO: teaching a GPT from preferences
With direct preference optimisation, our fine-tuned model learns from pairs of its own stories, where the one that uses more of the words asked for is preferred, and we measure what it gains and what it costs.
PPO: reinforcement learning for a GPT
Our model learns from its own stories, scored as it writes them, with proximal policy optimisation, a critic and a penalty for drifting, and we measure it next to DPO.
LoRA: fine-tuning a GPT with 2% of its weights
We fine-tune our model on instructions again with LoRA, which freezes its weights and trains a small low-rank update beside them, and measure what it saves and what it learns against full fine-tuning.
RoPE: rotary position embeddings for a GPT
We replace our model's table of positions with rotary position embeddings, which turn each query and key by its position, so attention scores depend on how far apart two tokens are, and the KV cache can slide past the context window.
BPE in C++: a byte-level tokenizer for a GPT
We write a byte-level BPE tokenizer in C++, test it against a Python reference, pretrain our model on the stories in BPE, and compare the two tokenizers' models in bits per character.
Distributed training: a GPT on four GPUs
We pretrain a model about six times bigger, in GPT-2 small's shape, on four RTX 4090s rented by the hour, with PyTorch's DistributedDataParallel, and measure how the speed scales from one GPU to four, and how much better the model gets in thirty minutes.
Flash attention in PyTorch, then in Triton
We write attention without its table of scores, first in PyTorch on the Mac, then as a Triton kernel on a rented NVIDIA GPU, forwards and backwards, and measure it against the FlashAttention-2 inside PyTorch.
Batching: serving a GPT to many users at once
We serve the model as an API in the shape of OpenAI's completions, measure it against a crowd of players from a game, write the stories for the requests that arrive together as a batch, and run it in Docker, with vLLM as the alternative.
Text diffusion: a GPT that writes all at once
We turn our GPT into a masked diffusion model, which writes a story by unhiding it, train it on the laptop and on four rented GPUs, start one from our next-token models' weights, and measure it against next-token prediction, comparing our code with LLaDA's, which trains this recipe at 8 billion parameters.
GRPO: reinforcement learning without a critic
Our model writes eight stories per instruction, and each is scored against the others, in place of PPO's critic, with group relative policy optimisation, the reinforcement learning behind DeepSeek-R1.
RMSNorm, SwiGLU, QK-norm: a modern GPT block
We bring our GPT's transformer block up to date with RMSNorm, SwiGLU, QK-norm and no biases, the changes recent open models made to it, each behind a config flag, with tests, and work out what they do to the parameter count.
Grouped-query attention: a smaller KV cache
We give our model grouped-query attention, where the query heads share their keys and values in groups, so the KV cache keeps fewer heads, and we check it against the attention we had and work out what it saves.
Pretraining on the web: a GPT overnight
We pretrain a model in GPT-2 small's shape on educational web pages for a whole night, on one RTX 4090 rented for under four dollars, and measure it against GPT-2 small on HellaSwag.
Tool use: a GPT that calls a calculator
We fine-tune the model we pretrained on the web to call a calculator, in plain text, and measure how often it calls it right, on questions asked the way training asks and in new ways, with the calculator and without it.
An agent with subagents, from a GPT we built
We fine-tune our web model into an agent that writes files and hands work to subagents, each a fresh context of the same model, and measure what the subagents buy.