This transcript comes from automatic captions and may contain small errors.
0:00 IntroThis is the story of how we turned a raw European language model into an assistant that can follow instructions, reason, and use tools.
The model is Open Euro LLM.
9 billion parameters trained on European languages with a context window of 256,000 tokens.
A base model like this only knows how to continue text.
Everything that makes it useful in a chat comes afterwards in post-training.
We will walk through every stage we ran on the Leonardo supercomputer.
What we did, what broke, how we found out, and what finally worked.
Post-training is a pipeline.
Each stage starts from the checkpoint of the stage before it.
First comes supervised fine-tuning.
We show the model millions of example conversations and train it to imitate the answers.
0:45 Chat templateWe did this three times.
Once for plain instructions, once for step-by-step reasoning, and once for tool use.
Then comes preference training called DPO, where the model learns from pairs of a better and a worse answer.
And finally, reinforcement learning, where the model tries things on its own and is rewarded when a program can check that the result is right.
Between every stage, we run the same evaluations, so we can see what each step adds and what it breaks.
Before any training, we have to decide exactly how a conversation looks to the model.
This is the chat template.
1:30 Instruction tuningSpecial tokens mark who is speaking, where an answer ends, where a hidden reasoning trace starts and stops, and where a tool call begins and ends.
Our template has a hard switch.
In default mode, the model answers directly.
In thinking mode, it first writes a reasoning trace between think tags.
The first lesson is the most boring one and the most expensive one.
Pin the tokenizer and the template with checksums and refuse to train if anything differs.
Stage one is instruction tuning.
We mixed 3.
1 million conversations, about 2.
4 billion tokens, with translations into 12 European languages.
The loss is only computed on the assistant's words.
The user's message is context, not something the model should learn to write.
2:15 Thinking modeTraining took 12 hours on 32 A100 GPUs.
The result follows instructions and answers in the user's language.
On our European benchmark, Europa, it scored 60%.
Stage two added a thinking mode.
370,000 reasoning examples replayed together with ordinary answers, so the default mode would not be forgotten.
Thinking mode turned out to be weak.
More than half of the reasoning traces never finished within 6,000 tokens.
Was the model just verbose?
We gave the unfinished ones 16,000 tokens.
81% still never stopped, and half of those were stuck repeating the same text.
3:00 Tool useSo, this was a termination problem, not a length problem.
More room does not help.
The model has to learn to stop.
Stage three taught tool use.
159,000 conversations, where the model calls functions, reads the results, or decides that no call is needed.
The first version looked fine during training.
Then we tested it, and every single tool call was broken.
The model opened a tool call, but never closed it.
The data was clean.
The cause was inside the model.
The new tag tokens started with almost identical output weights, and at a small learning rate, they never moved apart.
To the model, opening and closing a tool call looked like the same word.
The fix took 12 minutes on one GPU.
We refitted just eight rows of the output layer on the model's own hidden states.
The closing tag went from 0 to 99.
6% correct.
The lesson, a loss curve hides problems like this.
Check the special tokens after every stage.
How do we know if a stage helped?
We built an evaluation suite that runs automatically after every checkpoint.
Europa tests practical skills and alignment in European languages.
A quick reasoning set checks math in both modes.
A held-out tool check, code tests, and knowledge benchmarks complete the picture.
The evaluations found a surprising weakness.
4:30 Preference trainingAsked to answer only with a letter, the model wrote the letter and then explained itself anyway.
The cause was in the data.
99% of the training answers to strict format questions ignored the format.
The model learned exactly what we showed it.
Stage four was preference training.
For each prompt, there are two answers, one chosen and one rejected, and the model learns to prefer the chosen one more than a frozen copy of itself does.
We prepared 1.
4 million pairs.
The first run was healthy but slow, 25 seconds per step on eight nodes.
The full epic would have cost 20,000 of our budget hours.
The bottleneck was communication.
In every step, the GPUs gathered the full model three times.
5:15 Merging weightsAdding nodes made it slower, not faster.
Doubling the batch per GPU ran out of memory.
So, we trained on a well-chosen subset of 250,000 pairs instead at about a fifth of the cost.
Preference accuracy reached 85%, but the tool check failed.
Valid tool calls dropped from 99 to 90% with runaway calls.
All the preference pairs were plain chat without tools.
Preference training pushed the model towards longer answers, and tool use paid the price.
The fix was not more training, it was arithmetic.
We averaged the weights of the tool model from stage three and the preference model half and half.
6:00 Reinforcement learningBoth come from the same parent, so their weights can be mixed.
The merged model got its tool skills back and kept most of the preference gains.
Europa went from 62% A merge takes minutes and needs no training.
It became one of our most useful tools.
Now the main event, reinforcement learning with verifiable rewards.
For each prompt, the model writes eight different answers.
A program checks each one.
Is the math answer right?
Does the code pass the tests?
Is the tool call correct?
Answers that score better than their groups average are made more likely and worse ones less likely.
No human labels and no reward model, just checks.
One rule matters a lot in thinking mode.
If the reasoning trace never closes, the reward is zero, even when a correct answer is hidden somewhere inside the loop.
We did not start with one big run.
We ran small experiments on two nodes, each answering one question.
The first probe worked, fewer runaway traces and higher accuracy.
But it also showed something unexpected.
Before RL, a quarter of the reasoning traces for non-English questions were written in the question's language.
After 48 updates, only 1% were.
The model moved its thinking to English, where it was strongest.
Rewarding the trace language brought it back, but cost accuracy.
So we chose to require only the final answer in the user's language and let the model think where it thinks best.
7:30 Combined RL runAnother lesson, we rewarded bare answers in default mode and the reward stayed at zero.
In almost 3,000 tries, the model never once gave a bare answer.
RL can only reinforce what the model already does sometimes.
So we changed the reward to give partial credit to short, correct answers, a ramp the model can climb.
Then we combined everything into one run, starting from the merged model.
Math in European languages, code and strict format questions.
Watch the training curves.
At the start, 41% of the answers were cut off at the token limit.
By the end, almost none.
The average answer shrank from 3,400 tokens to about 1,200, while the reward went up.
On held out tests, thinking accuracy rose from 43 to 62% and unfinished traces fell from almost half to 15%.
Code improved too, from 19 to 35% on Human Eval Plus.
For the first time, thinking mode beat direct mode.
The combined run did far better than any single experiment.
Skills learned together supported each other.
In parallel, we ran tool calling RL on multi-turn conversations, rewarding valid JSON, the right function names, and the right arguments.
Valid JSON reached 99 and a half percent, and the right function names improved, the best tool results of all our models.
We published this checkpoint openly with a model card that documents every stage, every data set, and every evaluation.
But the evaluations also showed regressions.
Grounding and safety dropped from 70 to 58% and refusal calibration fell as well.
Being honest about regressions is part of the process.
They tell us what to fix next.
A lot of the work had nothing to do with learning.
Our first RL attempts failed on job scheduling, on memory during weight updates, and on an optimizer that refused to run in offload mode.
Each fix cost one short test run.
Canceling one job silently canceled the jobs chained after it.
And once, a continuation job started with nothing to continue.
On a full cluster, big jobs wait for hours, while small one GPU jobs slip into the gaps.
Splitting evaluations into small jobs made everything faster.
So far, we have used about 15,000 of our 50,000 budget hours.
Training continues.
A second combined RL run adds instruction following and a penalty for overly long answers.
We are testing whether separate specialist runs, merged afterwards, can match one joint run.
To fix the safety regressions, a local judge model scores the model's own answers and the best and worst become new preference pairs.
10:30 ClosingAnd we are building new tests in the spirit of the artificial analysis index.
Questions where saying I don't know beats guessing and real document tasks like invoices and contracts.
So, what did we learn about post training?
Pin your templates, check your special tokens, evaluate after every step with enough tokens.
Data teaches exactly what it contains, including bad habits.
And RL can only reinforce what the model already does sometimes.
Run small experiments first, then combine them.
And when training breaks something, merging can often bring it back.
Post training is not one big run.
It is a loop of measuring, fixing, and measuring again.