Chapter times for this film are approximate.
0:00 SurpriseImagine a coin that always lands heads. Flip it, and you learn nothing. Now imagine a fair coin. Every flip tells you something.
Information is surprise. The rarer an event, the more it tells you, and Shannon measured that as the negative logarithm of its probability. A one in two event carries one bit. A one in eight event carries three. A one in a thousand event carries about ten.
Entropy is the average surprise of a source. Add up each outcome's probability times its surprise, and you get the entropy: H equals minus the sum of p times log p.
Take a weather station with four outcomes, with probabilities one half, one quarter, one eighth, one eighth. The entropy is one point seven five bits. And that is not just a number. It is the smallest average number of yes or no questions needed to identify the outcome, and the best code reaches it exactly.
1:02 The toolkitEntropy is largest when you are most uncertain. For a coin, it peaks at exactly one bit when heads and tails are equally likely, and falls to zero as the outcome becomes certain. A coin that lands heads seventy percent of the time carries only point eight eight bits.
Two variables can share structure. Mutual information is the entropy you save about one by knowing the other. In this table, each variable alone carries one bit, together they carry one point seven two, so knowing one tells you about point two eight bits about the other.
Now suppose the world follows a distribution p, and your model believes q. The cross entropy, the average surprise you suffer by using the wrong model, equals the true entropy, plus the Kullback Leibler divergence. The first term is the irreducible uncertainty in the world. The second is the price of being wrong. Learning is the act of shrinking the second term.
2:06 PhysicsThis is the same mathematics that Boltzmann and Gibbs wrote for physics. Entropy counts the microscopic arrangements compatible with what you can see. A gas spreads out, because there are overwhelmingly more ways to be spread out.
And information is physical. Landauer showed that erasing a single bit costs at least Boltzmann's constant times temperature times the natural log of two, about three zeptojoules at body temperature. A living brain stays organised by burning around twenty watts, and exporting the disorder as heat.
2:39 The prediction machineFrom this view, a brain is a machine that keeps its own surprise low. It holds a generative model, predicts the signals it will receive, and the prediction error is what it learns from.
Friston's free energy principle says the brain minimises a quantity that sits above its surprise. Free energy equals the surprise, plus a divergence term that can never be negative. And the long run average of surprise is exactly the entropy of the world it senses.
3:11 The entropic brainIn twenty fourteen, Robin Carhart Harris and colleagues proposed the entropic brain hypothesis: that the richness of a conscious state tracks the entropy of spontaneous brain activity.
Ordinary adult waking consciousness, which they call secondary, is constrained. The default mode network, a set of midline regions that includes the medial prefrontal and posterior cingulate cortices, acts like a governor, holding brain dynamics in a narrow set of states. Primary consciousness, as in dreaming, early psychosis, and the psychedelic state, is less constrained, and so has higher entropy.
They measured it in several ways: the variability of blood oxygen signals, the metastability of networks, which is the variance of their synchrony over time, the Shannon entropy of that distribution, and the entropy of connectivity motifs. Four limbic regions give sixty four possible connectivity graphs at each moment, so the ceiling is six bits.
Under psilocybin, blood flow and activity dropped in the hubs of the default mode network, its internal connections weakened, its usual opposition to attention networks relaxed, and alpha power in the posterior cingulate fell, in proportion to how much people felt their sense of self dissolve. Meanwhile metastability, and the repertoire of connectivity motifs, expanded.
4:31 CriticalityWhy should that matter? Physics offers a hint. Systems poised at a critical point, between order and chaos, show maximal metastability, maximal sensitivity, and cascades of activity of every size. The authors suggested the waking brain sits a little below that point, and primary states move it closer. Too little entropy is rigid. Too much is disorder. The best cognition lies on an inverted U.
4:57 Entropy is not enoughBut noise has maximal entropy, and no one thinks static is conscious. What matters is entropy together with integration. A system of independent coins has six bits of entropy and no structure. A system that always agrees has structure and almost no entropy. Complex systems, brains among them, need both: many possible states, tightly coordinated.
That is why the best clinical measure compresses a response. The perturbational complexity index sends a magnetic pulse into the cortex, records the spreading echo, and measures how compressible it is, using Lempel Ziv complexity, normalised by the source entropy. Awake and dreaming brains score high. Deep sleep and anaesthesia score low, with a boundary near zero point three one. And in spontaneous MEG, signal diversity rises under LSD, psilocybin, and ketamine.
5:50 Entropy in language modelsNow consider a language model. Its training loss is exactly a cross entropy: the average surprise of the true next token under its predictions. By the identity we saw, that equals the entropy of language itself, plus the divergence between the model and the data. Training pushes down the second term. The first is the floor.
That is why scaling curves flatten. The loss falls as a power law in model size and data, towards a constant, an irreducible entropy of text. Shannon estimated English at about one bit per character. Perplexity, the exponential of the loss, is the effective number of choices the model hesitates between at each step.
At inference, entropy becomes a dial. Dividing the logits by a temperature reshapes the distribution. Near zero it collapses onto the single most likely token. At high temperature it flattens toward uniform. Sampling is a choice between exploiting what the model knows, and exploring.
Reinforcement learning adds a reward, but anchors the model to the original with a KL penalty, a leash measured in the same units. A known side effect is entropy collapse: as the policy sharpens on what is rewarded, diversity falls. Maximum entropy methods push back, by paying the policy to stay uncertain.
Entropy can also catch a model that is guessing. Semantic entropy samples many answers, groups those that mean the same thing, and measures uncertainty over meanings, not words. When the answers scatter, the model probably does not know.
7:28 Order and surpriseBoth kinds of mind face the same trade. Early in life, and early in training, high entropy is how you explore and find structure. Later, constraint is how you act reliably. Brains anneal, and so do models, from flexible to constrained.
Some cautions. Entropy is not consciousness. It is a property of a signal, and depends on how you measure it. The entropic brain is a hypothesis, supported by some data, not proven. And a language model's entropy measures its uncertainty about text, not whether it experiences anything.
What remains is a single idea, seen from several sides. Intelligence, and perhaps consciousness, lives between order and surprise. Enough entropy to learn. Enough structure to know.