Module 2 of 7
Transformers
The model behind modern AI systems, built up from tokens: tokenization and embeddings, attention with masks, the full block with its parameter and FLOP counts, and the scaling laws that size models and datasets.
A decoder-only transformer turns text into tokens, tokens into vectors, and runs those vectors through a stack of identical blocks. Each block mixes information across positions with attention and transforms each position with an MLP. Its parameter count and per-token FLOPs follow directly from a handful of config values, and scaling laws relate those numbers to the loss a training budget can buy.
The lessons implement byte-pair encoding, attention with causal and padding
masks, a parameter and FLOP counter for published configs, and the Chinchilla
compute-optimal arithmetic. Afterwards you should be able to estimate a model’s
size, training compute, and per-token cost from its config.json, and explain
why production models are trained on far more tokens than the compute-optimal
rule suggests.
Lessons
- 2.aEmbeddings and tokenizationHow does text become vectors, and why does the tokenizer change cost and behavior?10 min
- 2.bAttention from scratchWhat does attention compute, why are masks and scaling needed, and what does it cost?10 min
- 2.cThe transformer blockWhat is inside a transformer block, and how do you count its parameters and FLOPs from a config?10 min
- 2.dScaling laws and computeGiven a compute budget, how large should the model be and how many tokens should it see?9 min