Module 1 of 7
Foundations
The mathematics every training run repeats: tensors and their shapes, gradients by backpropagation, the optimizers that use them, and how to tell learning from memorizing.
A training run is four ideas repeated billions of times. Data and weights are tensors whose shapes decide memory and arithmetic. Backpropagation computes the gradient of the loss with respect to every weight for about twice the cost of the forward pass. An optimizer turns gradients into updates, and its learning rate decides whether training converges or diverges. And validation tells you whether the model learned something general or only memorized its training set.
Each lesson builds one of these from scratch in standard-library Python. Afterwards you should be able to read a layer as shapes and bytes, explain where the factor of 6 in “6ND” training compute comes from, choose and debug a learning-rate schedule, and read a train/validation curve.
Lessons
- 1.aTensors and shapesWhat does a tensor shape tell you about memory, compute, and correctness?9 min
- 1.bGradients and backpropagationHow does a framework compute gradients for billions of weights, and what does it cost?10 min
- 1.cOptimizers and learning ratesHow do SGD, momentum, and Adam differ, and why does the learning rate decide whether training diverges?9 min
- 1.dGeneralization and overfittingHow do you tell a model that learned from one that memorized, and what moves the gap?9 min