Module 3 of 7
GPU architecture
What is inside a modern data-center GPU: streaming multiprocessors, warps, the memory hierarchy, and tensor cores with their number formats.
A GPU trades the latency tricks of a CPU for thousands of simple lanes and a very wide path to memory. This module builds the mental model the rest of the course leans on: the roofline that says whether a kernel is limited by arithmetic or by bytes, how a streaming multiprocessor schedules warps, how far each level of memory is from the arithmetic, and what tensor cores and low precision formats actually buy.
Afterwards you should be able to take a spec sheet and a kernel’s FLOP and byte counts and say which resource bounds it, how much latency the chip must hide, and what precision is safe for which part of the computation.
Lessons
- 3.aWhy GPUs, and the rooflineWhy do neural networks run on GPUs, and how do you tell whether a kernel is limited by arithmetic or by memory?8 min
- 3.bSMs, warps, and SIMTHow does a GPU actually execute thousands of threads, and why do branches inside a warp cost so much?8 min
- 3.cThe memory hierarchyHow far is each level of GPU memory from the arithmetic, and how much parallelism does it take to keep HBM busy?7 min
- 3.dTensor cores and number formatsWhat do tensor cores compute, and which number formats are safe for which parts of a model?8 min