Module 7 of 7
Post-training and evaluation
Adapting a pretrained model with fine-tuning and LoRA, aligning it with preference data, and measuring the result without fooling yourself.
Pretraining produces a model that continues text. Post-training turns it into something useful: supervised fine-tuning on demonstrations, then preference optimization so it prefers the answers people prefer. Both are cheap compared with pretraining, and both are easy to get subtly wrong.
This module works through the arithmetic of full fine-tuning against LoRA, the loss behind RLHF and DPO, and the ways benchmark numbers mislead, starting with test data that leaked into training.
Lessons
- 7.aFine-tuning and LoRAWhat does it cost to adapt a pretrained model, and what does LoRA trade away to make it cheap?10 min
- 7.bPreference optimizationHow do preference pairs turn into a better model, and what goes wrong?9 min
- 7.cEvaluation and contaminationHow do you know a benchmark score reflects ability rather than memorized test data?9 min