Subject 06 / Course
Machine learning systems
From gradients to clusters: the mathematics of training, transformers, GPU architecture and performance, and training at scale.
0 of 30 read
Large models are built from a few simple operations repeated at enormous scale. This course follows them from a single gradient to a training run across thousands of GPUs: what the mathematics asks for, what the hardware can deliver, and where the two meet.
It assumes you can read Python and remember a little linear algebra. It does not assume a machine-learning background.
What you need#
- Python 3.10 or newer. Programs use only the standard library and run offline; GPU behavior is simulated with arithmetic.
The modules
Foundations
The mathematics every training run repeats: tensors and their shapes, gradients by backpropagation, the optimizers that use them, and how to tell learning from memorizing.
Transformers
The model behind modern AI systems, built up from tokens: tokenization and embeddings, attention with masks, the full block with its parameter and FLOP counts, and the scaling laws that size models and datasets.
GPU architecture
What is inside a modern data-center GPU: streaming multiprocessors, warps, the memory hierarchy, and tensor cores with their number formats.
GPU performance
How GPU code is written and made fast: the CUDA programming model, coalesced access and occupancy, tiling, fusion and IO-aware kernels, and reading a profiler.
Training at scale
What changes when one training run spans thousands of GPUs: feeding data, numerical stability, data, tensor, and pipeline parallelism, sharded optimizer state, and surviving failures.
Clusters and hardware
How GPUs are wired together, how collectives move data across that wiring, how one GPU is shared, and how to compare accelerators from their datasheets.
Post-training and evaluation
Adapting a pretrained model with fine-tuning and LoRA, aligning it with preference data, and measuring the result without fooling yourself.