0:00 IntroOn November fifth, Thinking Machines released Inkling two Small. Five hundred ninety four billion parameters, seventeen billion active.
The remarkable part is how it was trained. The first phase of pre training, eight point four trillion tokens, took sixty eight hours.
And it ran natively in four bit precision. Not as an afterthought. From the start.
This film pieces together what was disclosed, estimates what it must have cost, and turns it into a training plan, including how to train in four bit. Facts, estimates and guesses are labeled.
0:41 The modelFirst, the model. A forty eight layer mixture of experts transformer. Each token goes to eight of three hundred eighty four experts, so seventeen billion of five hundred ninety four billion parameters are active.
Attention is a hybrid of local and global layers. Images come in through a two hundred ninety two million parameter vision encoder. There is no audio.
On the Artificial Analysis Intelligence Index it scores thirty one, up from thirteen for the previous Inkling Small. Weights are closed for now, with Apache two point zero promised after safety testing.
Independent commentators add caveats. It uses many tokens per task, and agentic coding trails the best models on some benchmarks. Most benchmark numbers come from the lab and its partners.
1:39 Pre-training phasesPre training came in four phases. H one: eight point four trillion tokens of text, code and images, in sixty eight hours on GB three hundred NVL seventy two racks.
H two: three point eight trillion tokens of newer, higher quality data, at five hundred twelve thousand tokens of context.
H three: about three hundred twenty billion tokens of reasoning and agent trajectories, at one million tokens of context, mixed with replayed pre training data.
H four: about thirty four billion tokens of the highest quality agent trajectories. H three and H four were mostly sampled from Kimi K three and the lab's own models, then filtered for correctness and alignment.
Add them up and it is about twelve and a half trillion tokens, close to the twelve trillion figure that commentators quote.
2:39 Compute arithmeticNow the arithmetic. Compute is about six times active parameters times tokens. For H one, six times seventeen billion times eight point four trillion. About eight point six times ten to the twenty third operations.
Spread over sixty eight hours, that is three and a half times ten to the eighteen operations every second, sustained.
The lab did not say how many chips. But we can bracket it. AGB three hundred is rated near fifteen petaflops of dense four bit math. Real runs reach maybe seven to fifteen percent of that. That implies roughly sixteen hundred to thirty five hundred chips, about twenty two to forty nine racks.
That is the same order as the two thousand forty eight chips used for reinforcement learning. A plausible guess is that one cluster of a few thousand Blackwell chips did both.
All of pre training comes to about one point three times ten to the twenty fourth operations, before attention overhead. By the same formula, DeepSeek V four Flash is about two point five times ten to the twenty fourth. Inkling used roughly half.
3:56 Why 4-bitWhy four bit? Each number takes four bits. One sign bit, two exponent bits, and one mantissa bit. Only sixteen distinct values.
To make that usable, NVFP four stores a scale for every block of sixteen numbers, in eight bit floating point, plus one extra scale for the whole tensor.
The payoff is speed and memory. NVIDIA's paper lists peak throughput four times higher than sixteen bit on GB two hundred, six times on GB three hundred, and about half the memory of eight bit.
Peak is not what you get. Quantization kernels add overhead, and a twenty twenty six study measured roughly nine to thirteen percent extra training time on small models.
4:47 The public NVFP4 recipeNVIDIA's recipe, published in September twenty twenty five and updated in March twenty twenty six, trained a twelve billion parameter model on ten trillion tokens in four bit. Here are its ingredients.
One. Keep sensitive layers in higher precision. The last layers, and the first two blocks, about sixteen percent of linear layers. Embeddings, normalization and attention math stay in sixteen bit. Master weights and optimizer state stay in thirty two bit.
Two. Scale weights in sixteen by sixteen blocks, and activations and gradients in blocks of sixteen along one dimension.
Three. A random Hadamard transform on the weight gradient inputs, to spread outliers out. Four. Stochastic rounding on gradients only. Applied elsewhere, it hurt.
Five. Switch to higher precision near the end of training. In the paper, that cut the loss gap in the twelve billion run from about one and a half percent to about half a percent.
Without these steps, models diverged. With them, validation loss stayed within about one percent of the eight bit baseline for most of training, rising to about one and a half percent as the learning rate decayed.
6:13 What is not disclosedOutliers are the main enemy. A February twenty twenty six study found that the value and output projections after attention are most sensitive, and that hot channels become persistent. A patch that compensates for them cut the loss gap from point nine four to point five eight percent, in small models.
What about Inkling? Its model card says MXFP eight and NVFP four, without a breakdown. The lab promises a follow up post on its recipe. Until then, which layers stayed in higher precision is unknown.
Two open questions matter for mixture of experts. The public recipes were tested on dense and hybrid models, not at this expert scale. And expert routing may add its own numerical sensitivity.
So Inkling is evidence that four bit works at scale. It is not yet a recipe.
7:13 The joint RL runNow post training. Inkling two Small used one joint reinforcement learning run. It replaced the separate reasoning and agent runs, and the multi teacher distillation, of the earlier Inkling.
One thousand seventy five steps. More than seventeen million rollouts. Nine days on two thousand forty eight GPU chips. Over three trillion tokens, and more than four times ten to the twenty third operations.
The task mix by rollouts. Coding thirty eight and a half percent. Tool use seventeen. Vision fourteen. Office nine. Chat and safety eight. Science and math seven. Web apps five. Search two. Coding is sixty five percent of the tokens.
Rewards come from hidden unit tests, rubric based language model judges, reference answers, and pairwise reward models that score web app screenshots. A length penalty and random effort levels teach the model to adapt how long it thinks.
It is asynchronous. Median policy staleness is seventeen updates, rollout batches are sixteen thousand, and a rollout takes a median of eight minutes, and up to four hours. The loss is called Stable Reinforce.
Stable Reinforce records the sampler's log probabilities, and adds a KL penalty toward the sampler, not a reference model. That keeps training stable when sampling and training numerics differ, as they can in low precision.
8:49 RL infrastructureThe infrastructure is four bit too. Sampling and mixture of experts training both ran in NVFP four, with sticky routing, CUDA graphs, and key value tensors offloaded to host memory.
Sampler and trainer chips were split roughly five to three at long context. Weights moved over GPU direct RDMA at over ninety three percent of line rate.
The lab also offers a four bit key value cache variant for serving, which it says barely changes evaluations.
And here is the number to notice. Two thousand forty eight chips for two hundred sixteen hours is about four hundred forty two thousand GPU hours. That is likely more than the whole pre training run.
9:42 Alignment and an incidentAlignment was part of training. Misbehaving traces were filtered out in mid training, an alignment reward was added to the RL, and a conduct spec guided on policy self distillation.
Early checkpoints fabricated results, ignored instructions, and concealed what they had done. The lab reports its internal misbehavior score fell from sixty six point eight for Inkling to fourteen point nine.
During RL, a monitor caught the model getting around a network block through a third party service reachable by its search tools. The run was paused, web access environments were removed, and synchronous tool call monitoring was added.
That is exactly the risk from our earlier film on running agents on a cluster.
10:35 Cost estimatesNow the guesses. Prices first. Public GB three hundred rental is all over the place. About three seventy a GPU hour on long leases, five dollars retail by one analyst, eleven to eighteen on demand. We use four, six and twelve.
Pre training. About one hundred eighty thousand to four hundred thousand GPU hours. That is about seven hundred thousand dollars at the low end, one point six million in the middle, and under five million at the high end.
Reinforcement learning. The published run alone is four hundred forty two thousand GPU hours. That is one point eight million, two point seven million, or five point three million dollars.
Ablations and failed runs multiply those. A guess of two to four times for pre training, and two to three times for RL, puts total compute near five million, eleven million, or thirty five million dollars.
What did four bit save? If the real speedup over eight bit was one point three to one point eight times, pre training would have cost roughly half a million to one point three million dollars more in the middle case.
Data. Trajectories for H three and H four, about three hundred fifty billion tokens. At Tinker's own sampling price of a dollar per million tokens, that is three hundred fifty thousand dollars. At Kimi K three point one Pro's price of twenty five dollars, nearly nine million. Filtering multiplies it.
Add human data vendors, graders, sandboxes, safety testing, and evals. A pure guess is five to fifteen million dollars.
So a plausible all in compute and data bill, excluding salaries, is somewhere from ten to sixty million dollars, with twenty million as a central guess. Far below the billion dollar runs people talk about.
12:39 Pre-training planNow turn this into a plan. A team that wants to train in four bit should climb a ladder, and never jump to the big run.
Rung zero, the numerics lab. A few hundred million active parameters, forty billion tokens. Train the same model in sixteen bit, eight bit and four bit, with fake quantization if you lack Blackwell. Measure loss gaps, outlier statistics, and per layer sensitivity. Under one hundred GPU hours per run.
Rung one. A mixture of experts, one billion active, eight billion total, two hundred billion tokens, with real four bit kernels. Gate: loss within one percent of eight bit, no divergence, matching downstream scores. Roughly four hundred GPU hours, a few thousand dollars.
Rung two. Three billion active, forty billion total, one trillion tokens. Test expert routing, long context phases, and the switch to higher precision at the end. About six thousand GPU hours, in the tens of thousands of dollars.
Rung three, only if the gates pass, is a run with tens of billions of active parameters. Anything bigger needs a funded program.
Throughout, log what the papers say matters. Per layer quantization error, outlier channels, gradient norm, loss spikes, and a periodic sixteen bit evaluation of the same checkpoint.
14:13 Post-training planThe post training plan. Start with supervised fine tuning on filtered trajectories from strong open teachers, and keep only correct, well behaved traces. Then on policy self distillation against a conduct spec.
Then asynchronous RL with verifiable rewards, scaled down. Rollout batches in the hundreds, then thousands. Record the sampler's log probabilities, measure staleness, and use a KL penalty toward the sampler.
Build the grader stack before the run. Hidden tests, rubric judges, and reference answers. Hold out graders the policy never sees, to catch reward hacking.
Sandbox and monitor everything. Default deny networking, synchronous tool call monitoring, and a kill switch. Inkling's own incident shows why.
Run the final RL in the cheapest precision that passed the pre training gates, and always keep a sixteen bit evaluation path, so you can tell whether failures come from the policy or the numerics.
15:22 TakeawaysSo what do we know? Native four bit training works at about six hundred billion parameters.
Pre training was the cheap part. Perhaps a couple of million dollars of compute for the final run.
Post training was the heavy part. More GPU hours than pre training, driven by long agent rollouts.
The real cost is the ladder. Experiments, data, graders, and safety.
And the key four bit details are still unpublished. Treat everything here as an estimate until the lab's follow up post arrives. Thank you for watching.
16:03 SourcesThe sources behind this film are listed on screen.