Module 12 of 16
Building blocks
The shared components every AI platform ends up with: a gateway and router in front of models, caches, queues with batch lanes, and an evaluation service.
Once more than one application calls a model, the same four components appear whatever the product. A gateway and router decides which model serves each request and what happens when that model is slow or down. Caches avoid paying twice for the same work, and some of them can return a wrong answer while doing it. Queues and batch lanes decide who waits when demand exceeds the GPUs. An evaluation service says whether a change made the system better or worse.
Each lesson treats one component as a design problem with numbers: routing cost and quality, cache hit and false-hit rates, queue waits under priority and preemption, and the statistics an evaluation gate needs. By the end you should be able to specify these components for a platform and say where each one fails.
Lessons
- 12.aGateways and routersHow should a gateway decide which model serves a request, and what should it do when that model is slow or down?16 min
- 12.bCachesWhich caches belong in an AI system, what must each key contain, and when does a cache hit return a wrong answer?17 min
- 12.cQueues and batch lanesHow do interactive and batch work share one GPU fleet without interactive users waiting behind batch jobs or batch never finishing?14 min
- 12.dEvaluation servicesWhat does a shared evaluation service need so that its pass or fail on a change can be trusted?15 min