GRPO: reinforcement learning without a critic

Intro

0:00 Hello and welcome back to episode sixteen, where we're going to train our model with GRPO, the reinforcement learning behind DeepSeek's reasoning models, which scores each story against the other stories written for its instruction, with no critic. Let's begin with a quick look at where we're going to be by the end of the episode.

0:18 After ten minutes of GRPO, asked for kite, river and brave, our model writes about a brave boy at a river, with brave again and again, and no kite. Its stories use about fifty percent of the words an instruction asks for, up from about twenty-seven, and we're going to put that next to PPO's, run again on this laptop.

0:39 On the website of the series, you can find more information about the project. The code is open source on GitHub, tagged at each step of each episode, so that you can follow along, and the docs get a page on GRPO today. And each episode has a page here as well, with its chapters and its code.

0:59 Now, before we get to today's work, let's quickly go over where we left our model. Last time was a detour into diffusion, and today we're back to our next-token model, which writes a story one token at a time, fine-tuned on instructions in episode three. And we go back to reinforcement learning, where episode eight left it. There, our fine-tuned model wrote stories for instructions that ask for words, a reward scored each story by the share of those words it used, and PPO made the stories that did better than expected more likely. What was expected came from a critic, a layer we trained next to the model to guess the reward a story was going to get, from the model's channels at each token.

A group, in place of a critic

1:40 Group relative policy optimisation, GRPO, gets that expectation from the model's own stories instead. For each instruction, our model is going to write a group of eight stories, and the reward scores each one, with the share of the words asked for that it uses, as in episode eight, so that we can compare the two methods. A story that used more of the words than its group did on average gets a positive advantage, and one that used fewer, a negative one, with no critic to train. Then the update is PPO's clipped objective, with a penalty for drifting from the fine-tuned model, and the model writes again. We're going to train it for ten minutes with PPO's KL weight, and for ten minutes with no penalty at all, which is what DAPO, a recipe from ByteDance, does. Then we compare both with episode eight's PPO, run again on this laptop. All right, let's get started with where GRPO comes from.

Where GRPO comes from

2:35 GRPO comes from DeepSeekMath, by Zhihong Shao and others at DeepSeek, from 2024. Its abstract describes a model of seven billion parameters, trained for mathematical reasoning, and introduces GRPO as a variant of PPO that improves its reasoning while using less memory. Figure four shows why, with PPO on top and its four models: the policy, a reference, a reward model, and a value model, the critic. In their case, the critic is a language model about the size of the policy, trained alongside it. In GRPO, at the bottom, the policy writes a group of outputs, o one to o G, for each question, the reward model scores them, and group computation turns those scores into advantages, with no value model. Dropping the critic saves its memory and its training, while the policy writes several answers per question.

3:27 GRPO is also what trained DeepSeek-R1, from 2025, one of the first open models to reason step by step before it answers. For DeepSeek-R1-Zero, they ran GRPO from their base model, with no fine-tuning first, and a reward from rules, with no reward model: whether the final answer is right, and whether the reasoning is written between think tags. Figure one shows its accuracy on AIME, a maths competition, during training, going from about sixteen percent to seventy-eight, over thousands of steps. Our reward is scored by code too, by counting the words asked for, so the method carries over to our model, even though our stories are a long way from maths competitions.

The group's advantage

4:10 In episode eight, a token's advantage was the story's reward, R, minus V, the value the critic expected at that token. GRPO drops the critic, and for a question, or for us an instruction, q, the policy as it was at the start of the rollout writes G answers, o one to o G, and the reward scores them, r one to r G. G is eight for us, and sixty-four in DeepSeekMath. The advantage of answer i is its reward, minus the mean of the group's rewards, which plays the critic's part: an estimate of the reward to expect for that instruction, from eight tries at it. Dividing by the group's standard deviation puts each group on one scale, whether its rewards are close together or far apart. Each token of answer i gets that advantage, where PPO's critic gave each token an advantage of its own. And when all eight stories get one score, such as none of the words in any of them, the standard deviation is zero, and so are all their advantages. Such a group adds no gradient, and we're going to count those groups.

GRPO's objective

5:10 The objective is equation three of DeepSeekMath, and inside the brackets is PPO's clipped objective from episode eight. Rho is the ratio of a token's probability now to its probability when the story was written, clipped between zero point eight and one point two, and it multiplies the story's advantage. One over the length of the story averages each story's tokens first, and then one over G averages the stories, so a story counts once, however long it is. The KL penalty goes in the objective, per token, with the estimate we wrote in episode eight, e to the r, minus r, minus one, which DeepSeekMath uses too. Beta weighs it, at zero point zero four in DeepSeekMath. We're going to try zero point zero five, our PPO's, and zero, which drops the penalty. Even then, we keep the reference, to measure how far the policy drifts from it.

6:02 DAPO, by Qiying Yu and others at ByteDance Seed and Tsinghua University, from 2025, changes a few parts of GRPO to train long chains of reasoning, and takes a base model of thirty-two billion parameters to fifty points on AIME. First, it removes the KL penalty, and their section two point three gives the reason: a model learning to reason at length is expected to move a long way from the model it started as, so holding it close isn't needed. Then it adds dynamic sampling, for the groups whose answers all tie. When all of a question's answers are right, their advantages are zero, and the group adds no gradient. So DAPO samples more questions, and drops those whose answers are all right or all wrong, until the batch is full. And it averages the loss over all the tokens of a group, instead of over each answer first, as GRPO's average gives each token of a long answer less weight. Our PPO averaged over the tokens too, while for GRPO, we're going to follow DeepSeekMath.

GRPO in code

7:03 GRPO goes in a new file, grpo dot py, which borrows most of PPO's. It imports copy, for the reference, os, sys, textwrap and time, NumPy, torch, and TensorBoard's writer, with no nn this time, as there's no critic to build. From our own files, it takes the checkpoints and runs folders, with loading and saving checkpoints, and the data folder. From dpo dot py, it takes the demo instruction and words rate, and from evaluate dot py, load instructions and words used, our reward. Pick device chooses the GPU when there is one, as in our other scripts. From ppo dot py, it takes the clip, the epochs, the learning rate and the minibatch size, so that GRPO trains with PPO's settings, along with pack and token log probs. Then the most tokens a story gets, and load asks, from preferences dot py, and write, from sample dot py. Prompts is sixteen, the instructions per rollout, and group is eight, the stories per instruction. Sixteen times eight is a hundred and twenty-eight stories, as many as a rollout of PPO's, so that the two methods write as many stories for each update, which keeps the comparison between them fair. And we evaluate each ten rollouts, as PPO does. Rollout writes the stories, with no gradients, as writing them trains no weights. The rng's choice picks sixteen different instructions, without picking one twice. For each one, we stack its prompt eight times, so generate writes eight stories for it in one batch. It writes two hundred tokens for each, at a temperature of one, with no top k, as in PPO, since the update compares probabilities with the full distribution the tokens were sampled from. As all eight start from one prompt, they differ from each other by chance alone, which is what lets us compare them. Above it, the comment: a group, one instruction, written for eight times, so its stories can be scored against each other. Each row of out holds the prompt and then the story, so we take the tokens past the prompt's length. Each story is cut at its first end of text, which it keeps, so that stopping is part of what it learns, and its reward is the share of the requested words it uses, between zero and one. Group advantages is what replaces the critic, with a few lines of tensor arithmetic on the rewards, and no weights to train. View reshapes the hundred and twenty-eight rewards into sixteen rows of eight, a row per instruction, and the comment over it says what comes next: in place of a critic, each story's reward against its group's, in spreads above the group's mean. Each story's advantage is its reward minus its row's mean, divided by its row's standard deviation, plus a small number, so a row with no spread doesn't divide by zero. Tied marks the rows whose eight rewards all equal the first one, and the comment over it says why they matter: a group whose stories all tie has no way to tell them apart, so its advantages are zeros. Their advantages come out next to zero anyway, but rounding in the mean can leave them a little off, so masked fill sets them to zero. View flattens the advantages back into one per story, and tied comes back with them, so that we can count those groups. Main takes the minutes and the KL weight, with a folder per weight under grpo. We seed PyTorch and NumPy, load the fine-tuned model as the policy, and make a frozen deep copy as the reference. The policy goes into evaluation mode, and the comment gives the reason, as in PPO: dropout off, so the probabilities at the update match the rollout's. AdamW trains the policy's weights alone, as there's no critic to train with them. It uses PPO's learning rate, and no weight decay, as PPO does. Load asks reads the training instructions by their length in tokens, which PPO used to stack eight different instructions of one length. A group here repeats a single instruction, so any length will do, and we flatten them into one list, asks. The held-out instructions and the TensorBoard writer follow, with the run named after its KL weight. Evaluate is PPO's, as it was in episode eight. Words rate writes a story for each of the fifty held-out instructions and measures the share of the words they use, and write gives us the demo story, a hundred and twenty tokens for rain, cake and jump. Add scalar logs the share to TensorBoard, add text logs the story, and flush writes them to disk, so that we can follow the run as it goes. We print the share, and the story under it, indented, and save a checkpoint named after the rollout, so that we can measure the model at each point later on. The budget is the minutes, in seconds, and start is the time now. We start with an evaluation at rollout zero, before any training, and the loop runs until the time is up. Each time around, it writes a rollout, and packs it with ppo dot py's pack into one padded batch, with a mask over the stories' tokens. And it keeps the rewards in a tensor, on the device. Group advantages gives one advantage per story, and the comment says that each of the story's tokens is going to share it, while tied comes back with them. With no gradients, old is each token's log-probability under the policy that wrote it, which the ratio is going to compare against, and ref is the reference's, for the KL. Unlike PPO, there are no values to compute, as there's no critic. The update is PPO's two passes over the rollout. Rand perm shuffles the hundred and twenty-eight rows, and split cuts them into minibatches of sixteen, so each pass sees each story once, in a new order. New is the log-probabilities, with gradients, and the ratio is e to the new minus the old. A is the minibatch's advantages, and unsqueeze adds a dimension, so that a story's advantage multiplies each of its tokens. Surrogate is the clipped objective: the smaller of the ratio times the advantage, and the clamped ratio times the advantage. The comment over it says where it comes from: PPO's clipped objective, and the KL from the reference per token, as in ppo dot py. R is the log of the reference's probability minus the policy's, and per token is the surrogate, minus the KL weight times e to the r, minus r, minus one, masked to the story's tokens. The loss sums each story's tokens and divides by its length, then takes the mean over the stories, with a minus, as the optimiser minimises. The comment over it reads: a mean over each story's tokens, then over the stories, so a story counts once, however long. And the step is PPO's: zero the gradients, backward, clip them at one, and the optimiser's step. After the update, drift is each story's KL from the reference, as in PPO, summed over its tokens and averaged over the stories. Ties counts the tied groups, out of the sixteen, as a number. We log the reward, the KL and the ties to TensorBoard, under rollout, so that they plot side by side. We print them for each rollout, with the minutes so far, so that the terminal shows the run as it goes, and each ten rollouts, we evaluate. The end is PPO's too, so if the last rollout wasn't followed by an evaluation, there's one more, so the final model is measured, and then we close the writer, save the final model in the run's folder, and print a line with the rollouts and the minutes. Main runs with ten minutes and a KL weight of zero point zero five, by default. Last, evaluate dot py names a model after its folder, and PPO and GRPO both save to folders named after the KL weight, so their results would overwrite each other. A checkpoint saved along the way keeps its own name, as before, and for a final model, we look at its folder. If the folder's name starts with kl dash, the name is the folder above it, ppo or grpo, a dash, and the folder, and the comment above it gives an example: ppo slash kl zero point zero five is named ppo dash kl zero point zero five, while any other folder gives its name, as before.

Ten minutes of GRPO

15:03 Let's train for ten minutes with a KL weight of zero point zero five, PPO's. At rollout zero, the policy is the fine-tuned model, and its stories use about twenty-seven percent of the words on the held-out instructions. Each rollout prints the average reward, the KL from the reference, which starts at zero, and the groups that tied. Few groups tie, about four percent of them over the run, as eight stories rarely all score alike when there are three words to find. Each ten rollouts, the demo story for rain, cake and jump shows the change. By rollout twenty, cake and jump come back again and again, and by rollout thirty, rain does. After thirty-two rollouts, about forty-one hundred stories, the held-out stories use about forty-seven percent of the words.

15:50 Then ten minutes with no penalty at all, with a KL weight of zero. Without the penalty, the KL grows faster, to about forty-four per story over the last five rollouts. The demo story ends up about a cake, which comes back in almost each sentence, along with jump. After thirty-six rollouts, the held-out stories use about sixty-one percent of the words.

16:24 On a chart, the reward climbs in all three runs. GRPO's goes from about thirty-six percent of the words over its first five rollouts. Over the last five, it's about forty-seven with the penalty, and fifty-nine without it, against about forty-six for PPO. Each rollout scores its own instructions, so the reward moves up and down by several percent from one rollout to the next. The KL tells the three runs apart more clearly than the reward does. With the penalty, GRPO's ends at about fourteen per story, and PPO's at about nineteen, while without the penalty, it keeps growing, to about forty-four.

The four models, measured

17:01 Now evaluate dot py, on the fine-tuned model, PPO's and the two GRPO models. The words used go from twenty-seven percent for the fine-tuned model to forty-five with PPO, fifty with GRPO and the penalty, and fifty-eight without it. The loss on plain stories goes from two point three eight to two point four eight for PPO, two point four six for GRPO with the penalty, and two point five six without it. And the stories that end within two hundred tokens go from forty-eight percent to thirty for PPO, forty-two for GRPO with the penalty, and eight without it.

17:37 Side by side, GRPO with PPO's KL weight uses about five points more of the words than PPO. The fifty instructions ask for a hundred and fifty words in all, so that's a handful of words, from a single run of each, and another seed could close or widen the gap. So I'd call the two methods close at our scale. On plain stories, their losses rose by about as much, zero point zero eight for GRPO and zero point one zero for PPO. Without the penalty, GRPO uses the most words, and its loss rises the most, by about zero point one eight, while eight percent of its stories end on their own, as a longer story has more chances to use a word.

18:21 We give both GRPO models the instruction from episode eight, rain, cake and jump, with the seed we used there. With the penalty, the story is about a truck and a cake that jump, with cake in almost each sentence and no rain, and it ends on its own. Without the penalty, all three words are in, and the story repeats cake and wet until it runs out of tokens, with no end. Both are reward hacking, as we saw with DPO and PPO: the reward counts the words, so repeating them isn't penalised.

GRPO next to PPO

18:52 PPO learns what to expect with a critic, trained alongside the policy. Ours was a single linear layer, while InstructGPT's was a model of six billion parameters. GRPO gets that expectation from the group's mean instead, so there's no second model to train or to keep in memory, which is what DeepSeek was after at their scale. At ours, the critic was one layer, so dropping it saved very little. In exchange, GRPO writes several answers per question, eight for us and sixty-four in DeepSeekMath, to estimate the mean. And a group whose answers all tie adds no gradient. With our reward, that was about four percent of the groups. In maths, where an answer is right or wrong, DAPO saw more and more questions with all their answers right as training went on.

What's real, and what's scaled down

19:41 The algorithm is DeepSeekMath's: a group of answers per question, each scored against its group, with PPO's clipped objective and a KL penalty per token. Their reward came from a reward model, and DeepSeek-R1-Zero's from rules on whether the answer is right, where ours counts words, which is easy to game, as both our runs showed. Their model had seven billion parameters and wrote sixty-four answers per question, where ours has fourteen point six million and writes eight stories per instruction, for ten minutes on my laptop. They made a single update per rollout, so the ratio stays at one and the clip doesn't come into play, while we kept PPO's two passes, where it does.

20:20 Last, the docs get a page on GRPO, with the group's advantage, the objective next to PPO's, and the results of both runs next to PPO's. It goes in the list on the docs' home page, and at the end of the navigation. And the README has the command to train with GRPO, with a KL weight of zero point zero five.

Wrap-up

20:41 So, today our model learned from its own stories with GRPO. For each instruction, it wrote a group of eight stories, and each story's advantage was its reward against its group's mean and spread, with no critic. PPO's clipped objective made the stories that beat their group more likely, and a KL penalty kept the model near the fine-tuned one. With PPO's KL weight, the words used went from about twenty-seven to fifty percent, close to PPO's forty-five, and its loss on plain stories rose about as much as PPO's. And with no penalty, the model drifted furthest, to fifty-eight percent of the words, in stories that mostly ran on to the end of their tokens.

21:23 Let me know in the comments whether you'd pick GRPO or PPO for a model like ours, and which other rewards you'd like us to try with it. And if you'd like to follow along, subscribe to the channel. I hope you find this useful, and thanks for watching.