Course
All lessons
18 written lessons on how large language models are built and trained, about 2 hours of reading in total. Read them in order or jump to what you need. Each one links to the films that cover the same ground.
Part 1
The basics
What a model is, how it reads text, and how it learns.
01What a language model actually isA language model is a function that predicts the next piece of text. Everything else, from chat to code, is built on top of that one trick.✓02Tokens: how text becomes numbersModels read and write tokens, not words. Tokens decide what you pay for, how much fits in context, and why models stumble on spelling and arithmetic.✓03Attention and the transformerAttention lets each token look back at earlier tokens and decide which ones matter. Stacked many times, it is the core of every modern language model.✓04How a model learnsTraining is a loop that measures how surprised the model was by real text and nudges its parameters to be less surprised next time.✓
Part 2
Feeding the model
Data, scale and the memory a model needs.
05Pretraining dataFrontier models read tens of trillions of tokens, but most of the raw web is thrown away first. Quality, mixture and repetition shape what the model learns.✓06Scaling laws: how big, how longLoss falls smoothly as parameters, data and compute grow. Scaling laws turn that into a budgeting tool, and also show where the tool stops working.✓07Parameters, precision and memoryA parameter is a stored number, and every number costs bytes. Precision, gradients and optimizer state decide whether a model fits on one chip or needs a cluster.✓
Part 3
The machine room
Chips, parallelism and what goes wrong at scale.
08GPUs and parallelismA GPU is fast only when its arithmetic units stay fed. Training a large model means splitting it across thousands of GPUs in several ways at once, and paying for the communication.✓09Mixture of expertsA mixture-of-experts model stores many specialist sub-networks but sends each token to only a few. It gets the capacity of a huge model at the arithmetic cost of a much smaller one.✓10When training breaksAt frontier scale something is always failing. How labs survive broken chips, loss spikes, silent wrong answers and fragile low-precision numbers.✓
Part 4
After pre-training
Turning a text predictor into something useful, and running it.
11Supervised fine-tuningHow a raw text predictor becomes an assistant by studying examples of good answers, and why a small, clean dataset can beat a large, messy one.✓12Preference tuning and reinforcement learningHow models learn from comparisons and from checkable rewards: RLHF, DPO, verifiable rewards, group-based RL, and the sandboxes and reward hacking that come with it.✓13EvaluationHow we measure what a model can do, and why every measurement has a catch: contamination, saturation, judge bias and scores that cannot be compared across versions.✓14Distillation and quantizationTwo ways to make a model cheaper to run: teach a small model to imitate a big one, or store the same weights in fewer bits. What each one keeps, and what it costs.✓15Inference and samplingHow a trained model produces text: a fast reading phase, a slow one-token-at-a-time writing phase, a memory called the KV cache, and the sampling rules that pick each token.✓16What "open" means for AI models"Open" can mean downloadable weights, public code, released data or a full training recipe. These are different things, with different freedoms, and few releases include all four.✓
Part 5
End to end
Two long guides that follow one real model, Agora 9B, from raw text to a working assistant.
17How a base model is trained, end to endEvery step from raw text to a base model, followed through one real model, the OpenEuroLLM 9B that sits under Agora, with interactive demos of the loss, the optimizer, the schedule and long context.✓18How a base model becomes an assistant, end to endSeven stages of post-training followed through one real model, Agora 9B, with its data, recipes, costs and scores, and interactive demos of loss masking, preference tuning, weight merging and reinforcement learning.✓