Renaissance LabsLearn

Models · How a model is trained

What it takes to train a frontier model

Data, chips, failures and cost

15 minIntermediate13 chapters

A frontier model is a file of numbers, but producing it takes tens of trillions of tokens, tens of thousands of chips and a great deal of engineering. Using only 2026 reports (GLM-5, DeepSeek-V4, Kimi K2.5, a Meta fault-tolerance paper and Epoch AI data), the film follows data, scaling, hardware, failures, post-training and cost, and flags what is an estimate.

What to take away

  • Pre-training reads tens of trillions of tokens, and most raw web text is filtered out before it is used.
  • A common rule of thumb puts training compute at about six times the active parameters times the tokens.
  • At 100,000 GPUs a failure arrives roughly every 18 minutes, so fault tolerance matters as much as raw speed.
  • Post-training now runs on huge fleets of sandboxed environments, and reward design is part of the job.
  • None of the three reports gives GPU hours or dollar costs, so cost can only be estimated from the buildout.

Go deeper

Read the transcript65 lines, click any line to jump

Sources

Where a source has no link, it is cited from the film's research notes. Estimates and unverified claims are labelled in the film.

Keep watching