Subject 07 / Course
Inference engineering
How large models are served: prefill and decode, KV cache memory, batching, quantization, serving engines, vLLM internals, and running inference in production.
0 of 24 read
Serving a large model is a memory and scheduling problem before it is a model problem. Each generated token reads the full weights and the growing KV cache, so throughput, latency, and cost are set by how the server packs requests into GPU memory and time.
The course starts from the arithmetic of a single forward pass, builds up batching and paged KV memory, follows a request through a serving engine and through vLLM, and ends with distributed and production concerns: parallelism, disaggregation, SLOs, capacity, and fairness. It assumes you can read Python and know what a transformer is at the level of “layers of attention and MLPs”.
What you need#
- Python 3.10 or newer. Every program runs offline and simulates the GPU with arithmetic; nothing needs a GPU.
The modules
Inference mechanics
What one forward pass costs: prefill and decode, the size of the KV cache, and why generating tokens is limited by memory bandwidth rather than arithmetic.
Batching and scheduling
How a serving engine packs many requests into each GPU step: continuous batching, paged KV memory, chunked prefill, priorities, and preemption.
Making it cheaper
Three ways to get more tokens from the same GPU: store fewer bits, verify several guessed tokens per weight read, and skip prefill work already done.
Serving systems
What a serving engine is made of, how the main open engines differ in mechanism, and how tokens get from the engine to a client.
vLLM internals
One request through the vLLM V1 engine: processes and wires, the token-budget scheduler, the paged KV cache manager, model execution, and running it in production.
Distributed inference
Serving one model across many GPUs: how tensor, pipeline, and expert parallelism split the work and what each costs in communication, when to split prefill from decode, and how mixture-of-experts models are served.
Production inference
Running inference as a service: latency SLOs and goodput, capacity plans that hold at peak, autoscaling when a replica takes minutes to start, and sharing a fleet among tenants without one starving the rest.