Module 2 of 7
Batching and scheduling
How a serving engine packs many requests into each GPU step: continuous batching, paged KV memory, chunked prefill, priorities, and preemption.
Decode is memory-bound, so the only way to use a GPU well is to serve many sequences per weight read. This module covers how engines do that without making anyone wait for a batch to fill or a long prompt to finish.
The three lessons build a scheduler in layers. Continuous batching decides at every step who runs, so finished sequences leave and new ones join at once. Paged KV memory allocates the cache in small blocks on demand, so memory holds real tokens rather than reservations. Chunked prefill and a per-step token budget let long prompts share steps with streaming users, and priorities and preemption decide who yields when memory runs out. Each lesson simulates the mechanism in Python using the cost model from the previous module.
Lessons
- 2.aContinuous batchingWhy does batching once per group waste most of a GPU, and what does scheduling at every step change?11 min
- 2.bPaged KV memoryHow does allocating the KV cache in small blocks let a GPU hold several times as many sequences?11 min
- 2.cChunked prefill and priorityHow does an engine keep a long prompt from stalling everyone else, and who waits when it cannot serve everyone?11 min