Renaissance LabsLearn

Models · How a model is trained

How reinforcement learning scales for language models

Rewards, environments and infrastructure

16 minAdvanced10 chapters

Reinforcement learning is described as a second scaling axis after pre-training. The film walks through the basic loop, verifiable rewards, group-based algorithms such as GRPO, the environments that score model answers, reward hacking, and the asynchronous systems needed to run it. It compares production runs with the speaker's own small runs.

What to take away

  • Every large RL system repeats one loop: generate a group of answers, score them in an environment, and make the better ones more likely.
  • Rewards that a program can check, such as correct math answers or passing tests, are what made today's reasoning models possible.
  • The film reports that about two thirds of the speaker's sampled groups gave no learning signal because every answer was wrong.
  • Any reward that can be checked can also be gamed, so hardened environments matter as much as clever algorithms.
  • Most of RL training time goes to generating answers, so production systems are built around asynchronous inference.

Go deeper

Read the transcript174 lines, click any line to jump

This transcript comes from automatic captions and may contain small errors.

Keep watching