Renaissance LabsLearn

Models · How a model is trained

Inkling 2 Small: training in 4-bit

What was disclosed, what it cost, and a plan

16 minAdvanced15 chapters

Thinking Machines released Inkling 2 Small on November 5, a 594 billion parameter mixture of experts model trained natively in 4-bit. The film pieces together what was disclosed, estimates compute and cost, and turns it into a staged plan for 4-bit pre-training and post-training. Facts, estimates and guesses are labeled.

What to take away

  • The first pre-training phase, 8.4 trillion tokens, took 68 hours on GB300 racks, running in NVFP4 (a 4-bit format with block scales).
  • Public NVFP4 recipes keep sensitive layers in higher precision and use stochastic rounding on gradients only.
  • The joint reinforcement learning run used 2,048 GPUs for nine days, likely more GPU hours than all of pre-training.
  • The lab's own cost is not disclosed, so the film's roughly 10 to 60 million dollar total is an estimate, not a reported figure.
  • A team should climb a ladder of small 4-bit experiments with clear gates before attempting a large run.

Go deeper

Read the transcript71 lines, click any line to jump

Sources

Where a source has no link, it is cited from the film's research notes. Estimates and unverified claims are labelled in the film.

Keep watching