https://mvgpt.nenuadrian.com/2026-10-22 https://mvgpt.nenuadrian.com/about/ https://mvgpt.nenuadrian.com/ep001-architecture/2026-10-15https://i.ytimg.com/vi/Vff1s2GzI_o/maxresdefault.jpghttps://i.ytimg.com/vi/Vff1s2GzI_o/maxresdefault.jpgBuild a GPT from scratch in PyTorch | Minimum Viable GPT #1We're starting a series where we build mvGPT, a minimum viable GPT, from scratch in PyTorch, and take it through the training pipeline of a language model. In this first episode, we look at TinyStories, the data we start with, and at Attention Is All You Need, and go through our decoder-only architecture. We see where its 14.6 million parameters are and how a prompt flows through it to the next token, then write it in PyTorch, class by class.https://www.youtube.com/embed/Vff1s2GzI_o19002026-10-15gpttransformerpytorchlanguage modelllmattention is all you needself-attentionmulti-head attentiongelutinystoriesdeep learningmachine learningfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep002-pretraining/2026-10-15https://i.ytimg.com/vi/KA0aVTQ5Wak/maxresdefault.jpghttps://i.ytimg.com/vi/KA0aVTQ5Wak/maxresdefault.jpgPretraining a GPT from scratch in PyTorch | Minimum Viable GPT #2Today, we pretrain our GPT on TinyStories, starting with a word-level tokenizer and the next-token objective, with cross-entropy as the loss. We write the training loop, with AdamW, a learning rate that warms up and then follows a cosine, and gradient clipping, and train for ten minutes on a laptop. Then we write stories with the model and with an early checkpoint, and measure the checkpoints on held-out stories, from the loss and perplexity to the likeliest next tokens.https://www.youtube.com/embed/KA0aVTQ5Wak16032026-10-15gptpretrainingpytorchlanguage modelllmtokenizertinystoriescross entropyadamwlearning rate scheduleperplexitytraining loopdeep learningfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep003-sft/2026-10-15https://i.ytimg.com/vi/WUsVr66Bgq0/maxresdefault.jpghttps://i.ytimg.com/vi/WUsVr66Bgq0/maxresdefault.jpgSFT: fine-tuning a GPT to follow instructions | Minimum Viable GPT #3A pretrained model is an autocomplete, so in this episode we fine-tune ours to follow instructions, with TinyStories-Instruct, where each story comes with words to use, features and a summary. We mask the loss so the model learns the story and not the instruction, pad batches of different lengths, and fine-tune for ten minutes. Then we measure the model before and after, on how many of the requested words its stories use, and move the run's logs over to TensorBoard.https://www.youtube.com/embed/WUsVr66Bgq016092026-10-15gptsupervised fine-tuningsftinstruction tuningpytorchlanguage modelllmtinystoriesloss maskingcross entropyevaluationtensorboardmatplotlibdeep learningfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep004-kv-cache/2026-10-15https://i.ytimg.com/vi/JPlzOZouQxQ/maxresdefault.jpghttps://i.ytimg.com/vi/JPlzOZouQxQ/maxresdefault.jpgKV cache from scratch in PyTorch | Minimum Viable GPT #4Our model runs the whole story through all its blocks for each token it writes, so this time we give it a KV cache, which keeps the keys and values of the earlier tokens. We add it to attention, with the positions and the mask it needs, and check that it changes nothing, with matching logits and stories. On a laptop, it writes about four times faster on the CPU, and less so on the GPU, and we finish with what a KV cache costs at the scale of Llama 3.1.https://www.youtube.com/embed/JPlzOZouQxQ11982026-10-15gptkv cacheinferencegenerationattentionpytorchlanguage modelllmtransformerbenchmarkgpumpsgrouped-query attentionvllmfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep005-deploy/2026-10-16https://i.ytimg.com/vi/rn72Y_9dVE0/maxresdefault.jpghttps://i.ytimg.com/vi/rn72Y_9dVE0/maxresdefault.jpgDeploy a GPT: Hugging Face, Gradio, a browser | Minimum Viable GPT #5This time, we deploy our model, first to the Hugging Face Hub as a GPT-2 that transformers can load, with a tokenizer.json, an 8-bit ONNX export and a tag per episode. Then a Gradio app loads each version from the Hub and writes stories, on my machine and in Docker on a VPS, deployed over SSH. And a single web page runs the model in the browser with Transformers.js, downloading it when you ask for a story.https://www.youtube.com/embed/rn72Y_9dVE016592026-10-16gptdeploymenthugging facetransformerstransformers.jsgradioonnxquantizationdockervpspytorchlanguage modelllmgpt-2tokenizersafetensorsfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep006-profiling/2026-10-16https://i.ytimg.com/vi/7C0USj9KtI0/maxresdefault.jpghttps://i.ytimg.com/vi/7C0USj9KtI0/maxresdefault.jpgProfiling a GPT with PyTorch's profiler | Minimum Viable GPT #6In this episode, we look at where our model's time goes, starting with a graph of its 163 operations in Google's Model Explorer, exported with torch.export. Then PyTorch's profiler times a full pass and the writing of a token at a time, on the CPU and on the GPU. And we measure a few decisions in the code, such as our GELU written out, torch.compile and the size of the vocabulary, as each changes the speed by a different amount on each device.https://www.youtube.com/embed/7C0USj9KtI011002026-10-16gptprofilingpytorch profilermodel explorertorch.exporttorch.compileperformancegpumpscpuattentiongelupytorchlanguage modelllmfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep007-dpo/2026-10-16https://i.ytimg.com/vi/aQ4WxQfB2S4/maxresdefault.jpghttps://i.ytimg.com/vi/aQ4WxQfB2S4/maxresdefault.jpgDPO: teaching a GPT from preferences | Minimum Viable GPT #7In this episode, our model learns from preferences, with direct preference optimisation, DPO, which needs no reward model and no reinforcement learning. The fine-tuned model writes four stories per instruction, and code prefers the one that uses the most of the requested words to the one that uses the fewest. We train on 2,000 of these pairs for ten minutes, measure what changes, and run into reward hacking, as the model raises its score by writing longer stories.https://www.youtube.com/embed/aQ4WxQfB2S410382026-10-16gptdpodirect preference optimizationpreference optimizationrlhfalignmentfine-tuningpytorchlanguage modelllmtinystoriesreward hackingevaluationfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep008-ppo/2026-10-18https://i.ytimg.com/vi/8ewa70rUPME/maxresdefault.jpghttps://i.ytimg.com/vi/8ewa70rUPME/maxresdefault.jpgPPO: reinforcement learning for a GPT | Minimum Viable GPT #8This time, our model learns from its own stories as it writes them, with reinforcement learning and proximal policy optimisation, PPO, as in the PPO paper and InstructGPT. A reward scored by code counts the requested words, a critic learns what score to expect, and a KL penalty keeps the model near the fine-tuned one. We train for ten minutes with a small penalty and with a large one, and measure both against the fine-tuned model and DPO.https://www.youtube.com/embed/8ewa70rUPME13432026-10-18gptppoproximal policy optimizationreinforcement learningrlhfinstructgptkl divergenceadvantagepolicy gradientpytorchlanguage modelllmtinystoriesfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep009-lora/2026-10-16https://i.ytimg.com/vi/m5THIkAJJFk/maxresdefault.jpghttps://i.ytimg.com/vi/m5THIkAJJFk/maxresdefault.jpgLoRA: fine-tuning a GPT with 2% of its weights | Minimum Viable GPT #9We're going to fine-tune with low-rank adaptation, LoRA, which freezes the pretrained weights and trains a small update beside each linear layer, as the product of two thin matrices. We write it in PyTorch, fine-tune on instructions for ten minutes with about 2% of the weights, and merge the update back into the model. Then we compare it with full fine-tuning from episode three, on the file we keep, the optimiser's memory, the losses and the stories.https://www.youtube.com/embed/m5THIkAJJFk8982026-10-16loralow-rank adaptationpeftparameter-efficient fine-tuningfine-tuningadapterspytorchgptlanguage modelllmtinystoriesinstruction tuningfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep010-rope/2026-10-18https://i.ytimg.com/vi/CVeEPW67Gv8/maxresdefault.jpghttps://i.ytimg.com/vi/CVeEPW67Gv8/maxresdefault.jpgRoPE: rotary position embeddings from scratch | Minimum Viable GPT #10Our model learned a table of positions that stops at its context window of 256 tokens, so today we replace it with rotary position embeddings, from the RoFormer paper. Each query and key is turned by an angle that grows with its position, so attention depends on how far apart two tokens are. We pretrain for ten minutes next to episode two's model, and measure both past their context window, where our KV cache can now slide.https://www.youtube.com/embed/CVeEPW67Gv810432026-10-18gptroperotary position embeddingsposition embeddingsroformerkv cachecontext windowattentiontransformerpytorchlanguage modelllmtinystoriesfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep011-bpe/2026-10-18https://i.ytimg.com/vi/_o0mJYWdKQQ/maxresdefault.jpghttps://i.ytimg.com/vi/_o0mJYWdKQQ/maxresdefault.jpgBPE in C++: a byte-level tokenizer for a GPT | Minimum Viable GPT #11Our word-level tokenizer turns any word outside its vocabulary of ten thousand into an unknown token, so in this episode we replace it with byte-pair encoding, the tokenizer of GPT-2. We write a Python reference after the paper, then the algorithm in C++, built into a Python module with pybind11 and tested against the reference. Then we pretrain on the stories in BPE, time C++ against Python, and compare the models in bits per character, as their losses per token can't be compared.https://www.youtube.com/embed/_o0mJYWdKQQ18982026-10-18bpebyte pair encodingtokenizerbyte-level bpesubword unitsc++pybind11gpt-2tokenizationbits per characterpytorchgptlanguage modelllmfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep012-distributed/2026-10-18https://i.ytimg.com/vi/5-RJVvuGZyE/maxresdefault.jpghttps://i.ytimg.com/vi/5-RJVvuGZyE/maxresdefault.jpgDistributed training in PyTorch: 4 GPUs, DDP | Minimum Viable GPT #12For this episode, we rent four RTX 4090s on Vast.ai and pretrain a model about six times bigger than our laptop's, in GPT-2 small's shape, with PyTorch's DistributedDataParallel and torchrun. We try it on the laptop first, then measure the training speed on one, two and four GPUs, and train for thirty minutes on all of TinyStories. And we compare the result with the laptop's model, in bits per character and in its stories, and save it for Hugging Face.https://www.youtube.com/embed/5-RJVvuGZyE17262026-10-18distributed trainingdata paralleldistributeddataparallelddptorchrunncclmulti-gpurtx 4090vast.aigpu rentalpytorchpretraininggpt-2 smallgpt-neoxhugging facegptlanguage modelllmfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep013-flash-attention/2026-10-16https://i.ytimg.com/vi/HEYzYWquKos/maxresdefault.jpghttps://i.ytimg.com/vi/HEYzYWquKos/maxresdefault.jpgFlash attention from scratch: PyTorch, Triton | Minimum Viable GPT #13Attention as the paper writes it keeps a table of scores that grows with the square of the length, so in this episode we write flash attention, which computes it a block at a time. We start with attention by hand and its memory on the Mac, then write the online softmax in PyTorch, and then as a Triton kernel on a rented RTX 4090. And we measure our kernel against FlashAttention-2, which PyTorch already runs on NVIDIA GPUs, and pretrain the model with each.https://www.youtube.com/embed/HEYzYWquKos23462026-10-16flash attentionflashattentionflashattention-2tritongpu kernelcudaattentiononline softmaxpytorchscaled dot product attentionapple siliconrtx 4090gptlanguage modelllmfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep014-serving/2026-10-18https://i.ytimg.com/vi/6XGhd_83uwU/maxresdefault.jpghttps://i.ytimg.com/vi/6XGhd_83uwU/maxresdefault.jpgBatching: serving a GPT to many users at once | Minimum Viable GPT #14In this episode, we serve our model to many users at once, say the players of a game like nanoMMO from my other series, through an API in the shape of OpenAI's completions. Written one request at a time, the wait grows with the queue, so the server writes the stories of the waiting requests together, as a batch padded on the left. We measure it with a script that simulates a crowd of players, put the server in Docker, and look at vLLM as the alternative.https://www.youtube.com/embed/6XGhd_83uwU15152026-10-18llm servingbatchingdynamic batchingcontinuous batchingvllmopenai apicompletions apihttp serverpythonthreadingkv cachepaddingattention maskdockerpytorchtransformersgptlanguage modelllmfrom scratchmvgptnanommo https://mvgpt.nenuadrian.com/ep015-diffusion/2026-10-18https://i.ytimg.com/vi/LVWHstzgqE4/maxresdefault.jpghttps://i.ytimg.com/vi/LVWHstzgqE4/maxresdefault.jpgText diffusion: a GPT that writes all at once | Minimum Viable GPT #15This time, our GPT becomes a masked diffusion model, which writes a story by starting from a window of hidden tokens and unhiding the ones it's surest of, anywhere in the story. We write its loss and its sampler next to LLaDA's code, and train it for ten minutes on the laptop and for thirty on four rented RTX 4090s. Then we measure it against next-token prediction, fill in the middle of stories, and start a diffusion model from a next-token model's weights, as Dream 7B did.https://www.youtube.com/embed/LVWHstzgqE420182026-10-18diffusion language modelmasked diffusiontext diffusionlladamdlmdream 7bdiscrete diffusiondenoisinginfillingbidirectional attentionelbogpttransformerpytorchlanguage modelllmfrom scratchrtx 4090vast.aimvgpt https://mvgpt.nenuadrian.com/ep016-grpo/2026-10-18https://i.ytimg.com/vi/SoFl6r8Qz-0/maxresdefault.jpghttps://i.ytimg.com/vi/SoFl6r8Qz-0/maxresdefault.jpgGRPO: reinforcement learning without a critic | Minimum Viable GPT #16Today, we write group relative policy optimisation, GRPO, the method behind DeepSeekMath and DeepSeek-R1, which does reinforcement learning without the critic our PPO needed. The model writes eight stories per instruction, and each story's advantage is its reward measured against the group's mean. We train for ten minutes with a KL penalty and without one, on episode eight's reward, and measure both against the fine-tuned model and PPO.https://www.youtube.com/embed/SoFl6r8Qz-013032026-10-18gptgrpogroup relative policy optimizationpporeinforcement learningdeepseekdeepseek-r1deepseekmathreasoningadvantagekl divergencepytorchlanguage modelllmtinystoriesfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep017-modern-block/2026-10-18https://i.ytimg.com/vi/cPiQO4AilGY/maxresdefault.jpghttps://i.ytimg.com/vi/cPiQO4AilGY/maxresdefault.jpgRMSNorm, SwiGLU, QK-norm: a modern GPT block | Minimum Viable GPT #17Our transformer block still follows GPT-2's, so in this episode we bring it up to date with the changes open models made, from LLaMA to Qwen3. Following their papers, we write RMSNorm, SwiGLU, QK-norm and linear layers without biases by hand, each behind a config field, with tests against PyTorch's own. And we keep our earlier checkpoints loading as before, and work out the parameter count before and after.https://www.youtube.com/embed/cPiQO4AilGY21722026-10-18gpttransformertransformer blockrmsnormswiglugluqk-normqk normalizationlayernormsilullamapalmqwen3olmo 2attention logitspytorchlanguage modelllmfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep018-gqa/2026-10-18https://i.ytimg.com/vi/SpKMenz36TM/maxresdefault.jpghttps://i.ytimg.com/vi/SpKMenz36TM/maxresdefault.jpgGrouped-query attention: a smaller KV cache | Minimum Viable GPT #18Our KV cache keeps a key and a value for each of the six heads in each block, so this time we add grouped-query attention, as in Llama 2's bigger models and in Llama 3.1. Its query heads share their keys and values in groups, and we write it into our model as a config option, with tests against multi-head attention and against writing without the cache. With two key and value heads in place of six, the cache takes a third of the memory.https://www.youtube.com/embed/SpKMenz36TM13462026-10-18gptgrouped-query attentiongqamulti-query attentionmqamulti-head attentionkv cacheattentioninferencellama 2llama 3transformerpytorchlanguage modelllmfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep019-web/2026-10-18https://i.ytimg.com/vi/15vafg1wdpI/maxresdefault.jpghttps://i.ytimg.com/vi/15vafg1wdpI/maxresdefault.jpgPretraining on the web: a GPT overnight | Minimum Viable GPT #19In this episode, we move our model from children's stories to the web, with FineWeb-Edu, pages that a classifier scored as educational, and BPE merges learned on them. We work out how many tokens a night on one GPU can buy, against the Chinchilla paper's twenty tokens per parameter, and train a model of about 110 million parameters for nearly eight hours on a rented RTX 4090. In the morning, we measure it on HellaSwag next to GPT-2 small, and in bits per character on stories and on web pages.https://www.youtube.com/embed/15vafg1wdpI16032026-10-18pretrainingfineweb-edufinewebweb datacommon crawlchinchillascaling lawsgradient accumulationcheckpointinghellaswagbenchmarkgpt-2 smallrtx 4090vast.aigpu rentalbpetokenizerpytorchgptlanguage modelllmfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep020-tools/2026-10-18https://i.ytimg.com/vi/UrlqjWEuYH4/maxresdefault.jpghttps://i.ytimg.com/vi/UrlqjWEuYH4/maxresdefault.jpgLLM tool calling from scratch: a calculator | Minimum Viable GPT #20Our web model writes fluent text but can't do arithmetic, so today we teach it to call a calculator, as Toolformer did, writing the call as plain text for the program running it to fill in. We generate fifty thousand questions and fine-tune for ten minutes on the laptop, with the questions and the calculator's results left out of the loss. Then we measure the calls on held-out questions, worded the way training words them and in new ways, with the calculator and without it.https://www.youtube.com/embed/UrlqjWEuYH412502026-10-18tool usefunction callingcalculatortoolformerfine-tuningsftloss maskinggptlanguage modelllmfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep021-agent/2026-10-18https://i.ytimg.com/vi/9SYXGDioA6g/maxresdefault.jpghttps://i.ytimg.com/vi/9SYXGDioA6g/maxresdefault.jpgAI agent from scratch, with subagents | Minimum Viable GPT #21In this episode, our model becomes an agent, fine-tuned once into three roles: the agent, which hands parts of a task to subagents, a math role with the calculator, and a files role in a sandbox. Each subagent works in a fresh context, and its answer is all that goes back to the agent. We fine-tune for thirty minutes on the laptop, measure each role on held-out tasks, compare subagents with one long context, and put the agent in a Gradio chat that shows each step.https://www.youtube.com/embed/9SYXGDioA6g13602026-10-18agentsai agentssubagentstool usefunction callingsandboxorchestratorcontext windowfine-tuninggradiogptlanguage modelllmfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep022-long-context/2026-10-20https://i.ytimg.com/vi/2C2JYv2agzo/maxresdefault.jpghttps://i.ytimg.com/vi/2C2JYv2agzo/maxresdefault.jpgLong context LLM: stretching RoPE with YaRN | Minimum Viable GPT #22Our web model was pretrained on windows of 1,024 tokens, and its loss climbs past them, so in this episode we stretch its rotary embeddings four ways, from position interpolation to YaRN. We measure each on long web pages, by position, against a sliding window, and with the pass key test, a number hidden in filler. YaRN keeps the loss flat out to 4,096 tokens with no training at all, and then we fine-tune on long pages for fifteen minutes with two of the methods.https://www.youtube.com/embed/2C2JYv2agzo23302026-10-20gptlong contextcontext windowroperotary position embeddingsyarnposition interpolationntk-awarerope scalingcontext extensionpasskey retrievalkv cachefine-tuningpytorchlanguage modelllmfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep023-optimizers/2026-10-22https://i.ytimg.com/vi/0ysdFCrmdlU/maxresdefault.jpghttps://i.ytimg.com/vi/0ysdFCrmdlU/maxresdefault.jpgMuon vs AdamW, Lion and Sophia: a fair race | Minimum Viable GPT #23All our training scripts use AdamW, so in this episode we put it in a fair race with Lion, Sophia and Muon, each written from its paper, with tests. They train one model on one set of batches for a thousand steps, each at its best learning rate from a sweep, with three seeds. Muon reaches in 1,000 steps the loss AdamW needs about 1,840 for, while Lion and Sophia land behind a tuned AdamW.https://www.youtube.com/embed/0ysdFCrmdlU27322026-10-22gptoptimizeradamwadammuonlionsophianewton-schulzlearning ratehyperparameter tuningpretrainingpytorchlanguage modelllmfrom scratchmvgpt