mvGPT, a Minimum Viable GPT, is a series where I build a language model from scratch, to understand it end to end. We define a GPT in PyTorch and take it through the whole pipeline around it: pretraining, fine-tuning, reinforcement learning, evaluation and deployment. The focus is on how the blocks of parameters work together, in code you can read line by line, without the engineering tricks a model needs at scale.
The model starts out small enough to train in about ten minutes on a laptop. It begins by writing short children's stories, and from there we see how far it goes on a single machine, then on more than one, with other datasets and newer ideas from the research along the way.
Who it's for
It's for engineers and researchers who want to know what goes on inside a language model, rather than only call one. Some Python and PyTorch help. The maths comes in where the code needs it, next to the lines that use it.
The code
The code is on GitHub, under the MIT licence, with its docs here and the trained models on Hugging Face. Every step of every episode is a tag in the repository, so you can check out the code at any point in a video and run it yourself. Each episode's page lists its steps, with links to browse the code at each one and to see what changed since the step before.
How the series is made
Often, the voice you hear is generated with AI, and it's designed to sound like me. The wording follows how I speak, so none of it is left to chance. It's all highly moderated, and many hours go into designing the curriculum, informed by my own experience and research.
The videos themselves are built from code. The editor and the terminal you see are drawn from the repository's history and from recorded terminal sessions, while the browser pages are real captures.
Following along
New episodes go up on YouTube, and this site has an RSS feed for them. You can also browse the episodes by topic.