garagelmlearn/02 · teaching a model what to prefer
1. the problem: next-token prediction has no opinion
A pretrained base model is a text continuer. Ask it a question and it may answer, or it may continue with three more questions, because both are plausible ways for the internet to continue. Post-training installs behavior in two stages. Stage one, supervised fine-tuning (SFT): show it conversations and train the same next-token loss, but only on the assistant tokens, so it learns the shape of answering. Stage two, preference tuning: show it two answers to the same prompt, one preferred by humans, one not, and shift probability mass toward the preferred kind. This lesson is stage two, with stage one summarized just enough to follow the pipeline.
2. stage one in one paragraph, tax included
Our SFT run trained the 232M base on 310M tokens of SmolTalk conversations (38h). Chat quality, measured as masked validation loss on held-out conversations, improved from 2.073 (base) to 1.447. But the pre-registered capability gate failed: ARC-Easy dropped 2.2 points on the full test set. That is the alignment tax, measured: chat formatting was bought with a slice of science-question capability, while a smaller 60M-token SFT had paid roughly nothing. Rule of this lab: the failed gate is published next to the passed one.
3. the ingredients: two models, one dataset of arguments
DPO needs three things. First, a policy: the model being trained, initialized from the SFT checkpoint. Second, a frozen reference: an identical copy of the same checkpoint that never updates. Third, preference pairs: a prompt plus two responses, one marked chosen, one rejected. We used UltraFeedback (binarized), 53,952 training pairs of the form:
No new text is ever generated during training. Each step just scores existing responses: run all four combinations (policy and reference, each on chosen and rejected), and for each one sum the log-probability of the response tokens given the prompt. Four numbers per pair. Everything else is arithmetic on those four numbers.
4. the loss: rank the pair better than your past self
Write π_c and π_r for the policy log-probs of chosen and rejected, ref_c and ref_r for the reference log-probs. DPO computes a single logit per pair:
loss = −log σ(z)
Read it as a question: does the policy separate chosen from rejected by MORE than the reference already did? If yes, z is positive and the loss is small. If the policy still ranks the pair exactly like its frozen past self, z = 0 and the loss is exactly ln 2 ≈ 0.693, which is why every DPO run starts at 0.693: at step zero the policy IS the reference. The quantity β(π − ref) per response acts as an implicit reward, which is the paper's point: the language model is secretly its own reward model, and the RL loop of RLHF collapses into one classification-style loss. The frozen reference is the anchor that stops the policy from wandering into gibberish that happens to score well: it is only rewarded for reordering its preferences relative to where it started, and β sets how hard it is pushed.
5. play with the loss
Two knobs below stand in for the four log-probs: the policy gap (π_c − π_r) and the reference gap (ref_c − ref_r), in nats. Move them and watch z, the loss, and the gradient pressure respond. The presets replay the life of a training run.
The gradient-pressure row is the mechanism worth remembering: the update on each pair is scaled by σ(−z). Pairs the policy already ranks confidently right get almost no push; pairs it still ranks backwards get the full push. DPO spends its budget on its own mistakes. (In the real run these gaps are summed over whole responses, so they reach tens of nats; the shape of the loss is identical.)
6. running it on garage hardware
Lab rule: no multi-hour run launches without a sanity check written down first. For DPO the pre-registered gate was overfit 10 pairs: cycling them 12 times must drive the loss from 0.693 to below 0.1 with the reward margin strictly increasing every cycle. If a trainer cannot memorize 10 preferences, it will not learn 54k. It passed; the run launched.
Then the machine taught us something. The first launch ran out of memory at step 50 with a 57GB peak on a 43GB budget. The cause was new to us: preference pairs have wildly variable lengths, and thousands of unique tensor shapes fragment the MPS allocator until it hoards the whole machine. The fix was one idea: bucket every batch length to a multiple of 128, collapsing thousands of shapes into a few dozen. Memory fell to 43GB and, unexpectedly, training ran 1.9× faster (9.7s vs 18.4s per step), because the allocator stopped thrashing. One epoch over 53,952 pairs then took 9.0 hours, checkpointing every 200 steps. Constraints are not just obstacles; sometimes they are profilers.
7. eval: gates first, then the honest chat numbers
Two gates were pre-registered. G3, did it learn preferences? On 1,000 held-out pairs, the trained policy ranks chosen above rejected (by implicit reward) 65.2% of the time, past the 60% gate. G4, did it forget anything? Benchmarks vs the SFT model it started from:
| gate | metric | result | verdict |
|---|---|---|---|
| G3 preference accuracy | held-out ranking, 1k pairs | .652 (gate > .60) | PASS |
| G4 capability non-regression | HS / PIQA / ARC-E / WG vs SFT | −0.3 / −0.3 / −0.4 / −0.7 | PASS |
Unlike the SFT stage, DPO paid zero measurable capability tax. But gates are minimums, not the whole story. On the chat axes themselves, measured with the same harness for every model:
| model | IFEval prompt-strict | BoolQ | TruthfulQA-mc2 |
|---|---|---|---|
| 09 base | — | .623 | .393 |
| sft-full | 15.2% | .637 | .418 |
| dpo (shipped as chat v2) | 13.9% | .643 | .416 |
| SmolLM2-135M-Instruct (2T tokens) | 21.4% | — | — |
The uncomfortable row is IFEval: DPO slightly hurt rule-verifiable instruction following (13.9% vs 15.2% prompt-strict). That is not a bug in the run; it is a lesson about the objective. UltraFeedback teaches "sound like the answer people prefer," and IFEval scores "obey the literal constraint," and at 232M those pull in slightly different directions. A preference model is not automatically a more obedient model. Meanwhile TruthfulQA and BoolQ tilt gently toward the chat models, and preference ranking clearly works. The trade was judged worth shipping: this DPO endpoint is the published hybrid-gpt-232m-chat v2, with the regression disclosed on the model card and v1 still fetchable at revision v1.
8. what to take away
• Post-training at this scale is a usability lever, not a capability lever: chat quality moved a lot, benchmarks barely moved, in both directions.
• DPO is four log-probs and a sigmoid. The reference model, not RL machinery, is what keeps it stable, and every run starts at loss 0.693.
• The gradient concentrates on pairs the policy still ranks backwards; β sets how far from the anchor it may drift.
• Write the gates before the run. Publish the failed ones (SFT alignment tax) next to the passed ones (DPO's zero-tax pass).
• "Preferred" and "obedient" are different objectives. If you only remember one number from this page, make it 13.9 vs 15.2.