0:00 IntroA frontier model is, in the end, a file. A very large list of numbers, called weights.
But getting those numbers right takes months of work, tens of thousands of chips, a power plant's worth of electricity, and an enormous amount of text.
This film walks through what it really takes to train one. The data, the compute, the engineering, and the training that comes after the training.
Every number comes from a twenty twenty six source. The Z.ai GLM five report, the DeepSeek V four report, Kimi K two point five, a Meta paper on failures, and Epoch AI's data. We flag what is an estimate.
0:45 The five ingredientsThink of it as five ingredients. Data. Compute. An architecture. Engineering, to keep it all running. And post training, which turns a raw model into an assistant.
Pre training is the long, expensive stage. The model reads tens of trillions of tokens, and learns one simple game. Predict the next token.
Post training is shorter, but it shapes behavior. Whether the model follows instructions, reasons step by step, or uses tools.
Most of the compute goes into the first stage. Much of the behavior you notice is shaped in the second.
1:27 DataStart with data. A token is a piece of a word. GLM five trained its base model on a twenty seven trillion token corpus, and twenty eight point five trillion tokens across all stages. DeepSeek V four used more than thirty two trillion.
The raw material is mostly the web. Crawlers collect billions of pages, and then most of it is thrown away.
GLM five's report shows how. Web text goes through an embedding based quality classifier, and a second classifier for world knowledge. Code is de duplicated with fuzzy matching. Math and science are scored by a language model for educational value. And synthetic, templated text is filtered out.
Labs go beyond the web. A recent social media post about a university lecture by the chief scientist of Z.ai described six separate data engines. Web pages, scientific papers, books, code repositories turned into tasks with real tests, video, and audio transcribed by speech recognition.
Each source is its own engineering problem. Crawling, parsing, cleaning, removing duplicates, and judging quality. At this scale, data work is a large part of the job.
Data also changes over the run. DeepSeek V four filters out templated, auto generated web content, and adds agentic coding data in a mid training stage. The recipe is not one pass. It is a sequence.
3:06 Scaling and active parametersHow big should the model be, and how much data should it see? That is what scaling laws are for.
With mixture of experts models, the count that matters is the active parameters, the ones used for each token. GLM five has seven hundred forty four billion in total, but forty billion active. DeepSeek V four Pro has one point six trillion in total, and forty nine billion active.
A useful rule of thumb. Training compute is about six times the active parameters, times the tokens. For GLM five, that is six times forty billion times twenty eight point five trillion. About seven times ten to the twenty fourth operations. For DeepSeek V four Pro, close to ten to the twenty fifth.
Per active parameter, these models see a lot of data. About seven hundred tokens for GLM five. About six hundred seventy for V four Pro. And nearly twenty five hundred for the smaller V four Flash, with thirteen billion active parameters.
Context length is scaled in stages too. GLM five trains at thirty two thousand tokens, then one hundred twenty eight thousand, then two hundred thousand. DeepSeek V four goes from four thousand up to a million.
Million token context is expensive to serve, so efficiency is part of the design. DeepSeek reports that at a million tokens, V four Pro needs twenty seven percent of the compute per token of its predecessor, and ten percent of the memory cache.
4:47 Chips and powerNow the machines. The workhorse is the GPU chip. Kimi K two point five's report describes H eight hundred clusters, with eight four hundred gigabit network links per node.
The scale keeps growing. A January twenty twenty six paper from Meta and partners studies synchronous training on about one hundred thousand GPU chips, and ran a test on ninety eight thousand H one hundred chips.
Not everyone uses the same chips. The GLM five report describes support for seven Chinese chip platforms, including Huawei Ascend. The software has to adapt to each one.
Then there is power. Epoch AI estimates that the largest single site, xAI's Colossus two, draws about nine hundred fifty megawatts. The record has doubled about every ten months since mid two thousand twenty four, and sites near one and a half, then two gigawatts are projected.
In compute terms, Colossus two is estimated at about five hundred sixty thousand H one hundred equivalents in April twenty twenty six. Epoch projects sites of two to nearly four million by early twenty twenty eight.
6:06 ParallelismHere is the first engineering problem. The model does not fit on one chip. GLM five's seven hundred forty four billion parameters, stored as sixteen bit numbers, is about one and a half terabytes. DeepSeek V four Pro is more than three. One chip holds eighty gigabytes.
And training needs more than the weights. Gradients and optimizer state add several times more. All three of these models use the Muon optimizer, which tracks less state than Adam, but still adds terabytes.
So the work is split in several ways. Kimi K two point five reports sixteen way pipeline parallelism, sixteen way expert parallelism, and data parallelism with sharded optimizer state. GLM five adds context parallelism for long sequences.
All of this needs constant communication. DeepSeek V four uses a single fused kernel that overlaps computing, communication, and memory access, so chips do not sit idle waiting for the network.
Debugging at this scale is hard. So DeepSeek V four made its kernels batch invariant and deterministic, giving bitwise identical results. A bug you can reproduce is a bug you can fix.
7:29 When things breakNow the part nobody likes to talk about. Things break.
A January twenty twenty six paper from Meta and partners reports, for a thirty two thousand GPU job, two point three interruptions per thousand servers per day. Seventy eight percent were hardware related. Effective training time was ninety five to ninety seven percent.
Scale makes it worse. At one hundred thousand GPU chips, the paper estimates a failure every eighteen minutes. If every failure stops everything for up to ten minutes, effective training time falls to about forty four percent.
Their fix is to treat each data parallel replica as the unit of failure. Rebuild only the broken replica, while the rest keep training. In a test on ninety eight thousand H one hundred chips, a failure stalled training for about three minutes.
Silent data corruption is subtler. A faulty chip returns wrong numbers without any error message. Studies from twenty twenty six, including one from TU Berlin and an OSDI paper on production GPU faults, examine how it shows up in training.
Instability is the software side. DeepSeek V four used anticipatory routing, and clamped an activation inside its experts, to keep training stable. And its report admits the principles remain insufficiently understood.
9:03 Mixture of expertsArchitecture is the next lever. All three of the twenty twenty six models we are following are mixtures of experts.
Instead of every parameter working on every token, a router sends each token to a few specialist sub networks. Kimi K two point five has one point zero four trillion parameters, with three hundred eighty four experts, and eight active per token, so thirty two billion are used.
That cuts the compute per token, while keeping a huge capacity. The price is complexity. Balancing the load across experts, and moving tokens between chips.
Long context is the other pressure. GLM five uses sparse attention that picks the most relevant tokens. DeepSeek V four mixes compressed sparse attention with heavily compressed attention.
Numbers get smaller too. In post training, DeepSeek V four applies four bit quantization aware training to its expert weights, which cuts memory when serving.
One thing is missing. None of these reports gives GPU hours or a dollar cost. That used to be published. In twenty twenty six, it mostly is not.
10:22 Post-trainingA pre trained model is a powerful text predictor, but not yet an assistant. Post training changes that.
First comes supervised fine tuning. The model learns from examples of good answers, written by people or by other models.
Then reinforcement learning with verifiable rewards. GLM five uses GRPO for reasoning, then a separate asynchronous stage for agent tasks, so slow rollouts do not block training.
Environments are the new data. GLM five reports over ten thousand verifiable software engineering environments, across thousands of repositories and nine programming languages.
The sandboxes are huge. Kimi K two point five describes a rollout manager running up to one hundred thousand agent tasks at once. DeepSeek V four describes hundreds of thousands of concurrent sandboxes, including containers and Firecracker micro V Ms. The same isolation ideas as our last film.
DeepSeek V four then trains specialist models per domain, and merges more than ten of them into one student with on policy distillation, instead of mixing everything in a single run.
And rewards get hacked. Kimi K two point five added a reward term against spurious parallelism. GLM five fixed a slide generation exploit by fixing the renderer. Reward design is part of the job.
11:59 EvaluationBefore release, the model is measured on tasks that look like real work. DeepSeek V four Pro, in its maximum reasoning mode, reports eighty point six on SWE bench Verified, and sixty seven point nine on Terminal Bench two.
The test setup is itself an engineering system. Agent tests there allow up to five hundred steps and five hundred twelve thousand tokens of context.
Aggregate indexes tell a broader story. GLM five reports a score of fifty on the Intelligence Index version four, up from forty two for its predecessor, an average gain of about twenty percent.
Numbers are only part of it. Teams also read real transcripts, because benchmarks and rewards can both be gamed, and a model that scores well can still fail in strange ways.
12:56 CostSo what does it cost? The honest answer is that the labs do not say. The GLM five, DeepSeek V four and Kimi K two point five reports give no GPU hours or dollar totals.
What we can measure is the buildout. Epoch AI estimates that a typical one gigawatt AI data center needs about thirty eight billion dollars of up front capital, and about eight and a half billion dollars a year in total cost of ownership. Servers are about sixty percent of that.
It is now a macro number. Epoch reports AI related data centers, hardware and networking at about zero point eight percent of US GDP in early twenty twenty six, and computing infrastructure as a whole at about one and a half percent, double the earlier average.
And the final run is only part of it. Experiments, failed runs, data work, evaluation and salaries all sit on top.
Which is why small experiments matter. The GLM five team ran its attention ablations on a nine billion parameter base model, trained on one hundred ninety billion tokens, before committing to the full model.
14:15 What it takesSo, what does it take?
Data. Tens of trillions of clean, diverse tokens, and the pipelines to produce them.
Compute. Tens of thousands of chips, hundreds of megawatts, and networks that never rest.
Engineering. Parallelism, fault tolerance, deterministic kernels, and a lot of patience.
Research judgment. Small experiments, scaling curves, and the sense to know which results to trust.
And post training and evaluation, which turn raw capability into something useful and safe. None of these is enough alone. Thank you for watching.
14:59 SourcesThe sources behind this film are listed on screen.