https://mvgpt.nenuadrian.com/2026-10-20 https://mvgpt.nenuadrian.com/about/ https://mvgpt.nenuadrian.com/ep001-architecture/2026-10-15https://i.ytimg.com/vi/Vff1s2GzI_o/maxresdefault.jpghttps://i.ytimg.com/vi/Vff1s2GzI_o/maxresdefault.jpgBuild a GPT from scratch in PyTorch | Minimum Viable GPT #1We start a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model. We look at TinyStories, the data we're going to start with, and at Attention Is All You Need, and at what we change from it. Then we go through our decoder-only architecture twice: where its 14.6 million parameters are, and how a prompt flows through it to the next token. And we write it in PyTorch, class by class: the config, causal self-attention, the MLP with GELU written out, the transformer block, the GPT and its generate function. Then we print the model the way PyTorch sees it.https://www.youtube.com/embed/Vff1s2GzI_o19002026-10-15gpttransformerpytorchlanguage modelllmattention is all you needself-attentionmulti-head attentiongelutinystoriesdeep learningmachine learningfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep002-pretraining/2026-10-15https://i.ytimg.com/vi/KA0aVTQ5Wak/maxresdefault.jpghttps://i.ytimg.com/vi/KA0aVTQ5Wak/maxresdefault.jpgPretraining a GPT from scratch in PyTorch | Minimum Viable GPT #2To pretrain our GPT, we write a word-level tokenizer, download TinyStories and store it as token ids, and go through the next-token objective and cross-entropy. Then the training loop: random batches, AdamW with weight decay, a learning rate that warms up and then follows a cosine over the training time, and gradient clipping. We train for ten minutes on a laptop, write stories with the model and with an early checkpoint, try a high temperature, and measure the checkpoints on held-out stories: loss, perplexity, accuracy, and the five likeliest next tokens after "Once upon a time, there was a little". This is episode 2 of a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model.https://www.youtube.com/embed/KA0aVTQ5Wak16032026-10-15gptpretrainingpytorchlanguage modelllmtokenizertinystoriescross entropyadamwlearning rate scheduleperplexitytraining loopdeep learningfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep003-sft/2026-10-15https://i.ytimg.com/vi/WUsVr66Bgq0/maxresdefault.jpghttps://i.ytimg.com/vi/WUsVr66Bgq0/maxresdefault.jpgSFT: fine-tuning a GPT to follow instructions | Minimum Viable GPT #3A pretrained model is an autocomplete, so we fine-tune ours on TinyStories-Instruct, where each story comes with instructions: words to use, features such as a dialogue, a sentence the story should contain, and a summary. We parse the instructions into one order, mask the loss so the model learns the story and not the instruction, pad batches of different lengths, and fine-tune for ten minutes at a lower learning rate. Then we compare the model before and after, on how many of the requested words its stories use, how often they end on their own, and the loss on held-out stories. Last, we swap the run's JSON log for TensorBoard, and read its logs back in Python to plot them with matplotlib. This is episode 3 of a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model.https://www.youtube.com/embed/WUsVr66Bgq016092026-10-15gptsupervised fine-tuningsftinstruction tuningpytorchlanguage modelllmtinystoriesloss maskingcross entropyevaluationtensorboardmatplotlibdeep learningfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep004-kv-cache/2026-10-15https://i.ytimg.com/vi/JPlzOZouQxQ/maxresdefault.jpghttps://i.ytimg.com/vi/JPlzOZouQxQ/maxresdefault.jpgKV cache from scratch in PyTorch | Minimum Viable GPT #4Our model writes a token at a time, and runs the whole story through all its blocks for each one, so we give it a KV cache. As attention is causal, the keys and values of the earlier tokens don't change, so we keep them in a cache, a set per block, and run the new token alone. We add the cache to attention, with the positions and the mask it needs, and use it to generate. Then we check that it changes nothing, with matching logits, stories and evaluation numbers, and time it on a laptop's CPU and GPU: about four times faster on the CPU, and less on the GPU for one story, as with a model this small, most of a step on the GPU goes to launching its operations. Last, what a KV cache costs at the scale of Llama 3.1. This is episode 4 of a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model.https://www.youtube.com/embed/JPlzOZouQxQ11982026-10-15gptkv cacheinferencegenerationattentionpytorchlanguage modelllmtransformerbenchmarkgpumpsgrouped-query attentionvllmfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep005-deploy/2026-10-16https://i.ytimg.com/vi/rn72Y_9dVE0/maxresdefault.jpghttps://i.ytimg.com/vi/rn72Y_9dVE0/maxresdefault.jpgDeploy a GPT: Hugging Face, Gradio, a browser | Minimum Viable GPT #5A checkpoint loads with our own code and nothing else, so today we deploy the model: to the Hugging Face Hub, and to a VPS that hosts it and serves it. It has GPT-2's architecture, so we save its weights under GPT-2's names, write its word-level vocabulary as a tokenizer.json, and check that transformers loads it with our token ids and logits. We export it to ONNX, quantize it to 8 bits, and push it to the Hugging Face Hub, with a model card and a tag per episode. Then a Gradio app loads each version from the Hub and writes stories, on my machine, and in Docker on a VPS, deployed over SSH. Last, a single page runs the model in the browser with Transformers.js, downloading it when you ask for a story. This is episode 5 of a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model.https://www.youtube.com/embed/rn72Y_9dVE016592026-10-16gptdeploymenthugging facetransformerstransformers.jsgradioonnxquantizationdockervpspytorchlanguage modelllmgpt-2tokenizersafetensorsfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep006-profiling/2026-10-16https://i.ytimg.com/vi/7C0USj9KtI0/maxresdefault.jpghttps://i.ytimg.com/vi/7C0USj9KtI0/maxresdefault.jpgProfiling a GPT with PyTorch's profiler | Minimum Viable GPT #6We export the model with torch.export and look at it in Google's Model Explorer, as a graph of 163 operations. Then PyTorch's profiler shows where the time goes, on a full pass and while the model writes a token at a time, on the CPU and on the GPU, where the profiler times the CPU handing the work over. We colour the graph by each operation's time, and measure decisions in the code: our GELU written out, attention written by hand, torch.compile and the size of the vocabulary. Each changes the speed by a different amount on the CPU and on the GPU, so we measure both before we decide what to keep. This is episode 6 of a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model.https://www.youtube.com/embed/7C0USj9KtI011002026-10-16gptprofilingpytorch profilermodel explorertorch.exporttorch.compileperformancegpumpscpuattentiongelupytorchlanguage modelllmfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep007-dpo/2026-10-16https://i.ytimg.com/vi/aQ4WxQfB2S4/maxresdefault.jpghttps://i.ytimg.com/vi/aQ4WxQfB2S4/maxresdefault.jpgDPO: teaching a GPT from preferences | Minimum Viable GPT #7Supervised fine-tuning shows a model the answers we want, and preference optimisation shows it two answers and which is better. Our preferences are scored by code: the fine-tuned model writes four stories for an instruction, and the one that uses the most of the words asked for is preferred to the one that uses the fewest. Then direct preference optimisation, DPO, learns from 2,000 pairs, against a frozen copy of the model, with no reward model and no reinforcement learning. We measure the share of the requested words its stories use, and what that costs on held-out stories, and see reward hacking: the model raising its score with longer stories, which have more chances to use the words, whether or not they're better. This is episode 7 of a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model.https://www.youtube.com/embed/aQ4WxQfB2S410382026-10-16gptdpodirect preference optimizationpreference optimizationrlhfalignmentfine-tuningpytorchlanguage modelllmtinystoriesreward hackingevaluationfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep008-ppo/2026-10-18https://i.ytimg.com/vi/8ewa70rUPME/maxresdefault.jpghttps://i.ytimg.com/vi/8ewa70rUPME/maxresdefault.jpgPPO: reinforcement learning for a GPT | Minimum Viable GPT #8DPO learned from pairs of stories written ahead of time. Today the model learns from its own stories as it writes them, with reinforcement learning: each rollout writes 128 stories, a reward scored by code counts the share of the requested words they use, a critic learns what score to expect, and proximal policy optimisation, PPO, makes the stories that did better than expected more likely, with its clipped objective and a KL penalty that keeps the model near the fine-tuned one. We go through the PPO paper and InstructGPT, write it in PyTorch, train for ten minutes with a small penalty and a large one, and measure both against the fine-tuned model and DPO, along with what's real and what's scaled down. This is episode 8 of a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model.https://www.youtube.com/embed/8ewa70rUPME13432026-10-18gptppoproximal policy optimizationreinforcement learningrlhfinstructgptkl divergenceadvantagepolicy gradientpytorchlanguage modelllmtinystoriesfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep009-lora/2026-10-16https://i.ytimg.com/vi/m5THIkAJJFk/maxresdefault.jpghttps://i.ytimg.com/vi/m5THIkAJJFk/maxresdefault.jpgLoRA: fine-tuning a GPT with 2% of its weights | Minimum Viable GPT #9Full fine-tuning updates each of the model's weights and saves a full copy of it. Low-rank adaptation, LoRA, freezes the pretrained weights and trains a small update beside each linear layer of attention and the MLP, as the product of two thin matrices, B and A, with B starting at zero. We write it in PyTorch, fine-tune on instructions for ten minutes, and merge the update back into the weights, then measure it against full fine-tuning from episode three: the weights trained, the file we keep, the optimiser's memory, the losses and the requested words the stories use, along with what changes for models of billions of parameters. This is episode 9 of a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model.https://www.youtube.com/embed/m5THIkAJJFk8982026-10-16loralow-rank adaptationpeftparameter-efficient fine-tuningfine-tuningadapterspytorchgptlanguage modelllmtinystoriesinstruction tuningfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep010-rope/2026-10-18https://i.ytimg.com/vi/CVeEPW67Gv8/maxresdefault.jpghttps://i.ytimg.com/vi/CVeEPW67Gv8/maxresdefault.jpgRoPE: rotary position embeddings from scratch | Minimum Viable GPT #10Our model learned a table of positions, a vector for each of its 256 places, which stops at the context window and keeps its KV cache from sliding past it. We replace it with rotary position embeddings, from the RoFormer paper: each query and key is turned by an angle that grows with its position, so attention depends on how far apart two tokens are. We add them to the model as a config option, with a cache that slides, pretrain for ten minutes next to episode two's model, and measure both past their context window, on the CPU and the GPU of a laptop. This is episode 10 of a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model.https://www.youtube.com/embed/CVeEPW67Gv810432026-10-18gptroperotary position embeddingsposition embeddingsroformerkv cachecontext windowattentiontransformerpytorchlanguage modelllmtinystoriesfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep011-bpe/2026-10-18https://i.ytimg.com/vi/_o0mJYWdKQQ/maxresdefault.jpghttps://i.ytimg.com/vi/_o0mJYWdKQQ/maxresdefault.jpgBPE in C++: a byte-level tokenizer for a GPT | Minimum Viable GPT #11Our word-level tokenizer turns any word outside its ten thousand into an unknown token, and drops spaces and new lines. We replace it with byte-pair encoding, the tokenizer of GPT-2: a Python reference after the paper, then that algorithm in C++, with a heap of pair counts, built into a Python module with pybind11 and tested against the reference. The model gets a tokenizer on top of it, we pretrain on the stories in BPE, time the C++ against Python, and compare the word-level and BPE models in bits per character, as their losses per token can't be compared. This is episode 11 of a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model.https://www.youtube.com/embed/_o0mJYWdKQQ18982026-10-18bpebyte pair encodingtokenizerbyte-level bpesubword unitsc++pybind11gpt-2tokenizationbits per characterpytorchgptlanguage modelllmfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep012-distributed/2026-10-18https://i.ytimg.com/vi/5-RJVvuGZyE/maxresdefault.jpghttps://i.ytimg.com/vi/5-RJVvuGZyE/maxresdefault.jpgDistributed training in PyTorch: 4 GPUs, DDP | Minimum Viable GPT #12Our model has about fourteen million parameters and trains for ten minutes on a laptop, so this time we rent four RTX 4090s on Vast.ai and pretrain one about six times bigger, in GPT-2 small's shape, with data parallelism: PyTorch's DistributedDataParallel and torchrun, tried first on the laptop with gloo. We tokenize all of TinyStories on all of the rented machine's cores, measure the training speed on one, two and four GPUs, train for thirty minutes, and compare the result with the laptop's model in bits per character and in its stories. Then we save it for Hugging Face as a GPT-NeoX, with a byte-level BPE tokenizer.json. This is episode 12 of a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model.https://www.youtube.com/embed/5-RJVvuGZyE17262026-10-18distributed trainingdata paralleldistributeddataparallelddptorchrunncclmulti-gpurtx 4090vast.aigpu rentalpytorchpretraininggpt-2 smallgpt-neoxhugging facegptlanguage modelllmfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep013-flash-attention/2026-10-16https://i.ytimg.com/vi/HEYzYWquKos/maxresdefault.jpghttps://i.ytimg.com/vi/HEYzYWquKos/maxresdefault.jpgFlash attention from scratch: PyTorch, Triton | Minimum Viable GPT #13Attention as the paper writes it keeps a table of scores that grows with the square of the length, so we write it by hand and measure its memory and time on the Mac. Then flash attention's online softmax computes that attention a block at a time, first in PyTorch, then as a Triton kernel on a rented RTX 4090, forwards and backwards, tested against PyTorch's attention in float64. We find that PyTorch's attention already runs FlashAttention-2 on NVIDIA GPUs, measure our kernel against it, and pretrain the model with each. This is episode 13 of a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model.https://www.youtube.com/embed/HEYzYWquKos23462026-10-16flash attentionflashattentionflashattention-2tritongpu kernelcudaattentiononline softmaxpytorchscaled dot product attentionapple siliconrtx 4090gptlanguage modelllmfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep014-serving/2026-10-18https://i.ytimg.com/vi/6XGhd_83uwU/maxresdefault.jpghttps://i.ytimg.com/vi/6XGhd_83uwU/maxresdefault.jpgBatching: serving a GPT to many users at once | Minimum Viable GPT #14In a game such as nanoMMO, the MMO of my other series, many players could ask a storyteller for a story at once. So we serve the model from the Hub as an API in the shape of OpenAI's completions, with Python's own HTTP server and a thread for the model, and measure it with a script that simulates a crowd of players. One request at a time, the wait grows with the queue. Then the server writes the stories for the waiting requests together, as a batch padded on the left, with an attention mask and each row's own positions, tested against writing each prompt alone, and we measure it again, with batches of 16 and 64. The server goes into a Docker image of its own, and we look at vLLM, with continuous batching and PagedAttention, as the alternative, which runs on the CPU too. This is episode 14 of a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model.https://www.youtube.com/embed/6XGhd_83uwU15152026-10-18llm servingbatchingdynamic batchingcontinuous batchingvllmopenai apicompletions apihttp serverpythonthreadingkv cachepaddingattention maskdockerpytorchtransformersgptlanguage modelllmfrom scratchmvgptnanommo https://mvgpt.nenuadrian.com/ep015-diffusion/2026-10-18https://i.ytimg.com/vi/LVWHstzgqE4/maxresdefault.jpghttps://i.ytimg.com/vi/LVWHstzgqE4/maxresdefault.jpgText diffusion: a GPT that writes all at once | Minimum Viable GPT #15A masked diffusion model is the transformer we have, with attention both ways and one new token, [MASK]. It learns to fill in the tokens hidden in a window, from a few to nearly all of them, and it writes a story by starting from a window of [MASK] and unhiding the tokens it's surest of, a few at a time, anywhere in the story. We write the loss, a bound on the negative log likelihood, and the sampler, next to LLaDA's code, which trains this recipe at 8 billion parameters. We train it for ten minutes on the laptop and for thirty on four rented RTX 4090s, as episode twelve's model was, measure it in bits per character against next-token prediction, measure its loss at each share of tokens hidden, time it against the KV cache, fill in the middle of stories, and start a diffusion model from the weights of a next-token model, as Dream 7B did. This is episode 15 of a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model.https://www.youtube.com/embed/LVWHstzgqE420182026-10-18diffusion language modelmasked diffusiontext diffusionlladamdlmdream 7bdiscrete diffusiondenoisinginfillingbidirectional attentionelbogpttransformerpytorchlanguage modelllmfrom scratchrtx 4090vast.aimvgpt https://mvgpt.nenuadrian.com/ep016-grpo/2026-10-18https://i.ytimg.com/vi/SoFl6r8Qz-0/maxresdefault.jpghttps://i.ytimg.com/vi/SoFl6r8Qz-0/maxresdefault.jpgGRPO: reinforcement learning without a critic | Minimum Viable GPT #16In episode eight, PPO needed a critic to say what score a story should get. Group relative policy optimisation, GRPO, the method behind DeepSeekMath and DeepSeek-R1, does without one: the model writes a group of eight stories for each instruction, and each story's advantage is its reward against the group's mean, in units of the group's spread. We go through the DeepSeekMath and DeepSeek-R1 papers, write GRPO in PyTorch next to our PPO, train for ten minutes with a KL penalty and without one, on episode eight's reward, and measure both against the fine-tuned model and PPO, along with what's real and what's scaled down. This is episode 16 of a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model.https://www.youtube.com/embed/SoFl6r8Qz-013032026-10-18gptgrpogroup relative policy optimizationpporeinforcement learningdeepseekdeepseek-r1deepseekmathreasoningadvantagekl divergencepytorchlanguage modelllmtinystoriesfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep017-modern-block/2026-10-18https://i.ytimg.com/vi/cPiQO4AilGY/maxresdefault.jpghttps://i.ytimg.com/vi/cPiQO4AilGY/maxresdefault.jpgRMSNorm, SwiGLU, QK-norm: a modern GPT block | Minimum Viable GPT #17Our transformer block still follows GPT-2's, with a LayerNorm before each sublayer, GELU in the MLP, and biases in its linear layers and norms. We bring it up to date with the changes recent open models made, from LLaMA to Qwen3: RMSNorm, which divides each token's channels by their root mean square, with no mean and no bias; SwiGLU, an MLP with a gate, eight thirds of the embedding size wide; QK-norm, an RMSNorm on each head's queries and keys, which keeps the attention scores from growing with the weights; and no biases. We go through the RMSNorm, GLU variants, QK-norm, ViT-22B, small-scale proxies, LLaMA, PaLM and Qwen3 papers, write each change by hand behind a config field, test it, keep our earlier checkpoints loading as before, and work out the parameter count before and after, along with what's real and what's scaled down. This is episode 17 of a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model.https://www.youtube.com/embed/cPiQO4AilGY21722026-10-18gpttransformertransformer blockrmsnormswiglugluqk-normqk normalizationlayernormsilullamapalmqwen3olmo 2attention logitspytorchlanguage modelllmfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep018-gqa/2026-10-18https://i.ytimg.com/vi/SpKMenz36TM/maxresdefault.jpghttps://i.ytimg.com/vi/SpKMenz36TM/maxresdefault.jpgGrouped-query attention: a smaller KV cache | Minimum Viable GPT #18Our KV cache keeps a key and a value for each of the six heads in each block. With grouped-query attention, from the GQA paper and used in Llama 2's bigger models and in Llama 3.1, the query heads share their keys and values in groups. We go through multi-query attention, GQA and Llama 2's papers, draw our six heads three ways, and write it into our model as a config option, with tests against multi-head attention and against writing without the cache. With two key and value heads in place of six, the cache and the key and value projections take a third of the memory, which we work out from the config and print with summary.py. There's no training this time, just the code and the sizes, along with what's real and what's scaled down. This is episode 18 of a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model.https://www.youtube.com/embed/SpKMenz36TM13462026-10-18gptgrouped-query attentiongqamulti-query attentionmqamulti-head attentionkv cacheattentioninferencellama 2llama 3transformerpytorchlanguage modelllmfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep019-web/2026-10-18https://i.ytimg.com/vi/15vafg1wdpI/maxresdefault.jpghttps://i.ytimg.com/vi/15vafg1wdpI/maxresdefault.jpgPretraining on the web: a GPT overnight | Minimum Viable GPT #19So far, our models have learned from TinyStories, short stories for small children. This time, we train on FineWeb-Edu, web pages that a classifier scored as educational, with BPE merges learned on the web. We work out how many tokens a night on one GPU can buy, against the Chinchilla paper's twenty tokens per parameter, give pretraining gradient accumulation and a way to resume from a checkpoint, and train a model of about a hundred and ten million parameters for nearly eight hours, on one RTX 4090 rented on Vast.ai with the four dollars left on the account. In the morning, we measure it on HellaSwag, next to GPT-2 small and to the model we trained on stories, and in bits per character on stories and on web pages. This is episode 19 of a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model.https://www.youtube.com/embed/15vafg1wdpI16032026-10-18pretrainingfineweb-edufinewebweb datacommon crawlchinchillascaling lawsgradient accumulationcheckpointinghellaswagbenchmarkgpt-2 smallrtx 4090vast.aigpu rentalbpetokenizerpytorchgptlanguage modelllmfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep020-tools/2026-10-18https://i.ytimg.com/vi/UrlqjWEuYH4/maxresdefault.jpghttps://i.ytimg.com/vi/UrlqjWEuYH4/maxresdefault.jpgLLM tool calling from scratch: a calculator | Minimum Viable GPT #20Last time, we pretrained a model of 110 million parameters on educational web pages, overnight. It writes fluent text, but it can't do arithmetic. This time, we teach it to call a calculator, as Toolformer did: the model writes the call as plain text, the program running it computes the result and puts it into the text, and the model carries on. We generate fifty thousand questions, fine-tune for ten minutes on the laptop with the questions and the calculator's results left out of the loss, and measure the calls on held-out questions, worded the way training words them and in new ways, with the calculator and without it. This is episode 20 of a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model.https://www.youtube.com/embed/UrlqjWEuYH412502026-10-18tool usefunction callingcalculatortoolformerfine-tuningsftloss maskinggptlanguage modelllmfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep021-agent/2026-10-18https://i.ytimg.com/vi/9SYXGDioA6g/maxresdefault.jpghttps://i.ytimg.com/vi/9SYXGDioA6g/maxresdefault.jpgAI agent from scratch, with subagents | Minimum Viable GPT #21Last episode, our model learned to call a calculator. This time it becomes an agent: one model, fine-tuned once into three roles. The agent writes and lists files and hands the rest of a task to subagents, the math role with the calculator and the files role, which reads, writes and adds up files. Each subagent is the same model in a fresh context, and only its answer goes back. The files live in a sandbox whose limits the program enforces, whatever the model asks. We generate sixty thousand traces, fine-tune for thirty minutes on the laptop, measure each role on held-out tasks, and compare the agent with subagents against one context on a long task: right more often, with a far smaller context, for more tokens in all. Then we put it in a Gradio chat that shows each step. This is episode 21 of a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model.https://www.youtube.com/embed/9SYXGDioA6g13602026-10-18agentsai agentssubagentstool usefunction callingsandboxorchestratorcontext windowfine-tuninggradiogptlanguage modelllmfrom scratchmvgpt https://mvgpt.nenuadrian.com/ep022-long-context/2026-10-20https://i.ytimg.com/vi/2C2JYv2agzo/maxresdefault.jpghttps://i.ytimg.com/vi/2C2JYv2agzo/maxresdefault.jpgLong context LLM: stretching RoPE with YaRN | Minimum Viable GPT #22Our web model was pretrained on windows of 1,024 tokens. Its rotary embeddings let it read further, but past the window, its loss climbs, as some pairs of channels turn to angles it didn't see in training. We stretch the rotations four ways: position interpolation, NTK-aware interpolation, NTK-by-parts and YaRN, from the papers that introduced them, and measure each on long web pages, by position, against a sliding window, and with the pass key test, a number hidden in filler. YaRN keeps the loss flat out to 4,096 tokens with no training at all, and finds the key thousands of tokens back. Then fifteen minutes of fine-tuning on long pages, on the laptop, with position interpolation and with YaRN, and what a longer window costs in memory, along with what's real and what's scaled down. This is episode 22 of a series that builds mvGPT, a minimum viable GPT, from scratch in PyTorch, and takes it through the training pipeline of a language model.https://www.youtube.com/embed/2C2JYv2agzo23302026-10-20gptlong contextcontext windowroperotary position embeddingsyarnposition interpolationntk-awarerope scalingcontext extensionpasskey retrievalkv cachefine-tuningpytorchlanguage modelllmfrom scratchmvgpt