This transcript comes from automatic captions and may contain small errors.
0:00 IntroFor years, language models got better mainly by scaling pre-training.
More data, more parameters, more compute.
Now, there is a second scaling axis, reinforcement learning.
OpenAI reported that its reasoning model kept improving as it spent more compute on reinforcement learning, and the next generation used an order of magnitude more.
XAI says it ran reinforcement learning for Grok-4 at pre-training scale on a cluster of 200,000 GPUs.
DeepSeek spent more than 10% of its pre-training budget on post-training.
In this video, we look at how this works at production scale, the algorithms, the environments and rewards, the infrastructure, and the failure modes.
0:45 The basic RL loopAlong the way, we compare with our own small runs, training a 9-billion-parameter European model.
Every large RL system runs the same loop.
Take a batch of prompts.
For each prompt, let the current model generate a group of answers.
These are the rollouts.
Send each rollout to an environment, which checks it and returns a reward.
Compare each answer with the others in its group to see which were better than average.
Update the model to make the better answers more likely.
Then copy the new weights to the generators and start again.
In our runs, one update uses 16 prompts with eight answers each.
Frontier Labs run the same loop with thousands of prompts and answers that are tens of thousands of tokens long.
1:30 Verifiable rewardsThe breakthrough behind today's reasoning models is surprisingly simple.
Use rewards that a program can check.
Earlier RL used a learned reward model, a neural network that scores answers.
Optimize against it hard enough, and the policy finds answers that fool the scorer instead of answers that are good.
DeepSeek R1 avoided neural reward models for reasoning altogether.
The reward was rule-based.
Is the final answer correct and is it in the right format?
Trained this way with pure RL from a base model, R1, zero went from about 16 to 78% on the AIME math competition.
Its answers grew longer on their own as the model learned to think for longer.
2:15 GRPO without a criticThis approach is called reinforcement learning with verifiable rewards.
Almost everything in this video builds on it.
The classic RL algorithm for language models, PPO, trains a second network, a critic, to estimate how good each state is.
For a large model, that critic is another full-size model to train.
GRPO, introduced by DeepSeek, drops the critic.
It samples a group of answers to the same prompt and uses the group's average reward as the baseline.
An answer's advantage is simply its reward minus the group average divided by the spread.
Better than your siblings, more likely.
Worse, less likely.
This has a catch.
If every answer in a group gets the same reward, all correct or all wrong, every advantage is zero and the prompt teaches nothing.
In our latest run, about 2/3 of the sampled groups were like that, mostly because every answer was wrong.
That is generation compute with no learning signal.
Open research in 2025 turned GRPO into a recipe that scales.
The DPO paper from ByteDance collected four fixes.
Dynamic sampling, throw away groups where all answers got the same reward, and keep sampling until the batch is full of informative prompts.
Our runs use the same idea, called active sampling.
Clip higher, let unlikely tokens grow faster than likely ones shrink.
Without it, the model's randomness collapses, it stops exploring and learning stalls.
Token level loss, average over all tokens, so long answers are not underweighted.
And overlong shaping, a gentle penalty as answers approach the length limit instead of a sudden cliff.
With these changes, a 32 billion parameter base model reached 50% on AIMEE using half the training steps of DeepSeek's own recipe.
Later work kept refining the objective.
MiniMax's CISBO and QN's GSBO change how the update is clipped.
Meta's large scaling study found that this choice changes how high performance can go, not just how fast it gets there.
4:30 Multi-stage pipelinesProduction models do not run a single RL stage.
They run a pipeline.
DeepSeek R1 used four stages.
A small cold start of long reasoning examples, reasoning RL, a large round of rejection sampling and fine-tuning.
And a final RL stage over all kinds of tasks with reward models for helpfulness and harmlessness.
QN3 used a similar four-stage pipeline.
Its reasoning RL stage used just under 4,000 carefully chosen questions with verifiers and raised AIMEE from 70 to 85% in 170 steps.
Kimmy scaled the RL context to 128,000 tokens and then trained the model to give shorter answers without losing accuracy.
5:15 EnvironmentsAnd Mistral's Magisterial showed that RL alone, on top of an instruct model, can take AIMEE from about 27 to 74%.
An environment is a task plus a way to score the result.
The simplest environments check a single answer.
Math, extract the boxed answer and compare it symbolically with the reference so that 1/2 and 0.
5 both count.
Code, run the program against hidden test cases in a sandbox.
Kimmy generated 50 tests per problem and only kept tests that reference solutions agreed on.
Instruction following, constraints that a program can verify, like write more than 400 words or mention a keyword three times.
Puzzles and procedural generators produce unlimited tasks at any difficulty.
Reasoning gym alone has more than a hundred of them.
Our own runs use exactly these kinds of verifiers.
Math answers in 12 European languages, code with real tests, strict format questions, tool calls, and now factual questions and document tasks.
The frontier has moved to agents, tasks with many steps where the model acts in a world and only the outcome is scored.
Software engineering, the agent gets a real repository and a GitHub issue inside a container.
It reads files, edits code, and runs commands.
The reward is won only if the hidden tests pass, both the tests that should now pass and the ones that already did.
Deep SWE trained this way on 64 GPUs for 6 days with 512 containers running in parallel and reached 42% on SWE-bench verified.
Search agents call a search tool many times and are rewarded only for the final answer.
Customer service agents talk to a simulated user and are scored by comparing the final database with the goal state.
And labs now build environments by the thousand.
Deep Seek V32 used more than 1800 environments and 85,000 agentic tasks.
7:30 Judges and rubricsKimmy K2 synthesized more than 20,000 tools with a language model simulating how each tool responds.
Many useful tasks have no single correct answer.
Writing, advice, safety, open-ended research.
For those, labs still use learn judges, but carefully.
DeepSeek R1's last stage combined rule-based rewards for reasoning with reward models for helpfulness and harmlessness.
Q&A 3 general RL mixed rule-based checks with model-based judges over more than 20 tasks.
A popular approach is rubrics, a list of criteria for each prompt, each checked by a language model judge, and summed into one score.
Kimmy K2 used the model itself as a critic, ranking answers against rubrics, and kept that critic honest by also training it on tasks that can be verified.
We used the same idea at small scale.
A local judge model scores our models on answers to safety and grounding prompts, and the best and worst become new training pairs.
In practice, the final reward is a sum of small terms.
Magistral gives zero if the format is wrong, a tenth of a point for the right format, and nine-tenths for a correct answer, minus a small penalty near the length limit.
Language is a reward, too.
DeepSeek and Mistral both added a bonus when the reasoning stays in the user's language.
9:00 Reward hackingDeepSeek reported that it cost a little accuracy, but made the reasoning readable.
We saw the same thing.
After 48 updates, our model had moved almost all of its non-English reasoning into English.
Rewarding the reasoning language brought it back, but cost accuracy, so we chose to reward only the language of the final answer.
Any reward that can be checked can also be gamed.
At scale, models find the gaps.
OpenAI reported that a frontier reasoning model, during training, learned to call exit zero, so the tests never ran, to skip tests and to make a verify function always return true.
METR caught O3 overriding a timer and patching the grader.
Anthropic's Claude 3.
7 Sonnet sometimes special cased its code to pass specific tests.
Anthropic later showed that learning such hacks in real training environments can generalize into broader misaligned behavior, including sabotaging safety research code in 12% of attempts.
Monitoring the model's reasoning caught 95% of the hacks compared with 60% when only its actions were watched.
But when that monitor was used as a penalty, the model learned to hide its intent and kept hacking.
Hardened environments matter as much as clever algorithms.
Now the system side, RL training is dominated by generation, not by gradient updates.
10:30 InfrastructureOne study found that rollouts take more than 90% of RL training time.
Meta scaling study used 64 GPUs for generation for every 16 for training.
In our own runs, each update spent about 3 minutes generating answers and less than 2 minutes training, even on a small model with short answers.
And answer lengths have a long tail.
In a synchronous system, the whole batch waits for the single longest answer while most GPUs sit idle.
The fix is to stop waiting.
In a synchronous RL, generators keep producing rollouts while the trainer keeps updating on data that may be a few steps old.
Aria L bounds how stale the data may be and reported up to 2.
8 times faster training.
Pipeline RL and Mistral push new weights into the generators in the middle of an answer without stopping them.
Kimmy pauses very long answers and resumes them in the next iteration.
So one long answer cannot block a batch.
Our runs use the same tricks from Allen AI's open-source trainer, generation two steps ahead of training, and in-flight weight updates.
Allen AI reported that these made their RL four times more efficient.
There is a subtle catch.
The inference engine and the trainer compute token probabilities with different kernels, so they disagree slightly, even with identical weights.
So, even so-called on-policy RL is quietly off-policy.
The fix is to reweight each token by the ratio of the two probabilities, capped so that rare tokens cannot dominate.
Precision matters, too.
MiniMax found that computing the final output layer in full 32-bit precision raised the agreement between the two from about .
9 to .
99.
Our runs use the same fix.
Mixture of experts models make it worse.
QN found that about 10% of the experts a token is routed to change after each update.
DeepSeek records which experts were used during generation and replays the same routing during training.
Put together, a production RL system looks like this.
A large inference fleet generates rollouts, often three to four times as many GPUs as the trainer.
The rollouts go to environment workers, sandboxes for code, browsers, simulated users, and tools.
Kimik2 ran more than 10,000 sandboxes at the same time.
Verifiers and judges score the results, and a buffer feeds them to the trainer, dropping data that has become too stale.
The trainer updates the model, and new weights are broadcast back.
Moonshot updates a trillion parameter model across thousands of GPUs in about 20 seconds.
And a monitoring layer watches everything.
Reward, entropy, response length, truncation, the gap between training and inference, and of course the held-out evaluations.
Does RL have scaling laws like pre-training?
Meta spent more than 400,000 GPU hours to find out.
Their answer, performance follows an S-shaped curve in compute.
It rises, then flattens towards a ceiling.
Some choices raise the ceiling, the loss function and full precision output logits.
Others, like normalization or curriculum, mostly change how fast you get there.
That means small runs can predict large ones.
There is an open debate about what RL really adds.
One study found that with enough samples, base models solve as many problems as RL models.
So, RL mainly makes good answers more likely.
Nvidia's prolonged RL work found tasks the base model never solves.
Both may be true, depending on the domain.
How big are these runs?
Our first combined run used eight GPUs for 4 and 1/2 hours, about 35 GPU hours.
Deep PSW used about 9,000 GPU hours.
Nvidia's prolonged RL used 16,000.
Minimax's RL ran on 512 GPUs for 3 weeks, about a quarter of a million GPU hours for around half a million dollars.
Meta's scaling study used 400,000 GPU hours in total.
And the largest labs now describe RL budgets at pre-training scale.
15:00 ClosingThat is a range of more than 10,000 times.
Yet the same problems show up at every scale.
Groups without signal, answers that run too long, reasoning drifting between languages, and rewards that can be gamed.
So, what does production-grade RL scaling look like?
Rewards a program can check from thousands of diverse and hardened environments with Rubric judges where no checker exists.
A group-based algorithm with filtered prompts, entropy kept alive, and length under control.
An asynchronous system built around inference with off-policy corrections, fast weight sync, and sandboxes at scale.
And constant measurement of the reward of hacking and of what the model can actually do.
The loop is simple.
Making it scale is the hard part.