Module 6 of 7
Clusters and hardware
How GPUs are wired together, how collectives move data across that wiring, how one GPU is shared, and how to compare accelerators from their datasheets.
A training or serving job that spans more than one GPU runs at the speed of the slowest link it depends on. This module covers the links inside a server and between servers, the collective algorithms that move gradients and activations across them, the three ways to share one GPU between workloads, and how to read accelerator specifications without taking the marketing numbers at face value.
Afterwards you can predict the bottleneck link for a given placement, estimate how long an all-reduce takes, choose between time-slicing, MPS, and MIG, and compare accelerators on the metric that matters for your workload.
Lessons
- 6.aNVLink, PCIe, and topologyWhich link limits traffic between two GPUs, and how do you find out before the job is slow?11 min
- 6.bCollectives and networkingHow long does an all-reduce take, and when does a tree beat a ring?10 min
- 6.cSharing GPUsWhen several workloads share one GPU, which mechanism gives the throughput and isolation you need?11 min
- 6.dThe accelerator landscapeHow do you compare accelerators honestly from their published specifications?9 min