Models · Feeding the model · Lesson 5
Pretraining data
Frontier models read tens of trillions of tokens, but most of the raw web is thrown away first. Quality, mixture and repetition shape what the model learns.
A model can only learn from what it reads. For a frontier model, that is somewhere between 27 and 48 trillion tokens of text, code and now images and audio. How that pile is collected, cleaned and mixed has a large effect on the result, and labs publish less about it than about the architecture.
Where the text comes from
The largest source is the web. Crawlers collect billions of pages, and the raw HTML is turned into text by extraction: removing menus, ads and boilerplate and keeping the main content. Open projects such as FineWeb show this pipeline in public. Beyond the web, labs use books, scientific papers, code repositories and, for multimodal models, images, video and speech transcripts. The mixes are only partly disclosed.
- CrawlCollect billions of pages
- ExtractPull the main text from the HTML
- FilterKeep what looks useful
- DeduplicateRemove near copies
- MixChoose proportions by source
Quality filtering
Most of the raw web is poor: spam, machine-generated pages, navigation lists. Filtering keeps a small fraction. Typical tools are simple rules (length, symbols, repeated lines) and trained classifiers that score how much a page looks like text a person would want to learn from.
GLM-5's report describes several: an embedding-based quality classifier, a second one for world knowledge, and a language model that scores math and science for educational value. DeepSeek-V4 removes batches of templated, auto-generated web text, partly to avoid model collapse, where a model trained on machine output loses variety.
Deduplication
The web repeats itself. Pages are copied, quoted and scraped. If a passage appears a thousand times, the model sees it a thousand times, wastes capacity on it and may memorise it. Deduplication removes exact and near-exact copies, often with fuzzy matching that catches small edits. GLM-5 reports that its fuzzy deduplication of code produced about 28% more unique tokens than its earlier approach.
Mixing code, math and languages
Data is not one pile. The proportions of web, code, math, books and languages are tuned like any other design choice. Code and math are widely used as a core of the mixture, and DeepSeek-V4 reports a larger multilingual corpus and a focus on long documents such as papers and reports. The mix also changes over the run. Training often ends with stages that raise the share of high-quality data, longer contexts and agent-style data. DeepSeek-V4 trains at 4K, then 16K, 64K and finally 1M tokens of context. GLM-5 adds roughly 10 million issue and pull-request pairs, about 160 billion unique tokens of software engineering data, in mid-training.
Synthetic data
Synthetic data is text generated by models, then filtered or checked. It is most useful where answers can be verified, such as code with tests and math with known answers. One reported family, K2 Horizon, is said to use about 10T synthetic tokens in about 20T (a secondary-source figure, so treat it as uncertain). The risk is the one above: unfiltered machine text can narrow what the model learns.
When data runs short
If good text is finite, what happens when a run wants more tokens than exist? The paper Scaling Data-Constrained Language Models studied repeating data. Its finding, as commonly summarised, is that a few repetitions (around four passes) behave almost like fresh data, while many more give steadily smaller gains. That is why mixes use unique data first and repeat the best slices.
Contamination, licensing and openness
Contamination means benchmark questions leaked into training data, which inflates scores. Labs try to remove them, but with trillions of tokens it is hard to guarantee, so one evaluation score is weak evidence on its own.
Licensing is unsettled. Many models publish weights while keeping data undisclosed: Beam 1.0 is Apache 2.0 but its training data is not disclosed. Meta describes Muse Spark Open's data as public sources, crawls, third parties and its own products. Open weights and open data are different things, a theme taken up in the last lesson of the course.
Where to go next
For what a model does with this data, revisit How a model learns. Later lessons cover scaling laws and what "open" means. The data chapters of the frontier training and open models films show the same material in motion.