Renaissance LabsLearn

Minds · Where the two meet

Entropy: order and surprise

One idea in physics, brains and language models

8 minIntermediate9 chapters

Entropy is the average surprise of a source. The film builds the idea from coin flips, then follows it into physics, the free energy principle, the entropic brain hypothesis and criticality. It ends with language models, where training loss is a cross entropy, and with the limits of the idea.

What to take away

  • Information is surprise: a one in eight event carries three bits, and entropy is the average of that surprise.
  • Training loss for a language model is a cross entropy, equal to the entropy of language plus the divergence from the data.
  • Scaling curves flatten because the loss approaches an irreducible entropy of text.
  • Temperature reshapes the next-token distribution, trading off exploiting what the model knows against exploring.
  • The entropic brain is a hypothesis, and entropy is a property of a signal, not a measure of consciousness.

Go deeper

Read the transcript26 lines, click any line to jump

Chapter times for this film are approximate.

Sources

Where a source has no link, it is cited from the film's research notes. Estimates and unverified claims are labelled in the film.

Keep watching