Subject 05 / Course
Systems design
Invariants, failure, concurrency, scale, and data, then the same discipline applied to systems built around models: interfaces, retrieval, agents, evaluation, and AI infrastructure.
0 of 63 read
A system is a set of promises: this balance never goes negative, this token is used once, this request finishes or fails within two seconds. Design is the work of choosing those promises and then keeping them while requests overlap, machines fail, and load exceeds the plan. Security and reliability are the same discipline seen from two sides: both ask what still holds when something goes wrong.
The course starts from invariants and state machines, follows the ways systems fail, works through concurrency, consistency, scale, distributed data, and observability, and works four complete designs, each argued from its invariants rather than from a diagram of boxes. The second half applies the same discipline to systems built around a model, which is slow, expensive per call, non-deterministic, and wrong some of the time: model interfaces, retrieval, AI building blocks and infrastructure, agentic systems, evaluation in production, and five more worked designs. It assumes you can read Python and have built or operated a web service. It does not assume a distributed-systems background.
The path#
flowchart LR I["<b>Invariants</b><br/><small>what must hold</small>"] --> F["<b>Failure</b><br/><small>partial, slow, overloaded</small>"] F --> C["<b>Concurrency</b><br/><small>races, leases, consistency</small>"] C --> P["<b>Production</b><br/><small>SLOs, change, recovery</small>"] P --> S["<b>Scale and data</b><br/><small>queues, shards, consensus</small>"] S --> W["<b>Worked designs</b><br/><small>argued end to end</small>"] W --> A["<b>AI systems</b><br/><small>models, retrieval, agents, evals</small>"] W -.->|"incidents sharpen the invariants"| I class I neutral class C memory class P compute class W good class S queue class A compute classDef neutral fill:#f3f1ea,stroke:#5f5e55,color:#141413,stroke-width:1.5px classDef compute fill:#f5d195,stroke:#b0420c,color:#141413,stroke-width:1.5px classDef memory fill:#f5d195,stroke:#b0420c,color:#141413,stroke-width:1.5px classDef queue fill:#f5d195,stroke:#b0420c,color:#141413,stroke-width:1.5px classDef io fill:#f5d195,stroke:#b0420c,color:#141413,stroke-width:1.5px classDef default fill:#f5d195,stroke:#b0420c,color:#141413,stroke-width:1.5px classDef warn fill:#ef94ad,stroke:#b3154f,color:#141413,stroke-width:2.2px,stroke-dasharray:6 4 classDef good fill:#8dbb6f,stroke:#2b6a24,color:#141413,stroke-width:2.2px
How each lesson works#
- The idea in one minute. The claim, before the detail.
- A picture of the mechanism or the failure.
- How it really works, with the numbers and the named techniques.
- Code: a small Python simulation using only the standard library. Failures, clocks, and networks are simulated so every run is reproducible.
- Where it breaks. The assumption that, once violated, defeats the design.
- Remember this, Try it, and Check yourself.
What you need#
- Python 3.10 or newer and a terminal. Every program runs offline.
- A notebook for capacity math. Most designs fail on arithmetic before they fail on architecture.
The modules
Thinking in invariants
Before any boxes and arrows: what the system promises, which states it may be in, how much load it must carry, and how to choose between designs and write the choice down.
How systems fail
Distributed systems rarely fail cleanly. This module covers partial failure, retries, circuit breakers, overload, and blast radius, and the controls that keep one fault from becoming an outage.
Concurrency and consistency
What still holds when requests overlap, replicas lag, and processes pause: races, locks and fencing, consistency models, replication, and delivering effects exactly once.
Reliability in production
How to state reliability as a number, see what a running system is doing, change it without breaking it, and get the data back when something destroys it.
Worked designs
Four complete designs, each argued from its invariants: a rate limiter, a credential broker, a tamper-evident audit log, and a multi-tenant inference gateway.
Scale and performance
Why services get slow and how to make them fast again: queueing in pools and pipelines, caches, partitioning, load balancing against the tail, and profiling.
Distributed data
How replicated state agrees, how storage engines turn writes into files, how logs carry data between systems, and what it costs to commit one transaction across many machines.
Observability
The mechanics behind telemetry at scale: metric types and cardinality, histogram error, traces and profiles, sampling and its cost, alert delays, and the signals that matter on GPU fleets.
How AI systems differ
What changes when a model sits in the request path, and how to size an AI workload in tokens, seconds, and dollars before drawing any boxes.
Model interfaces
The contract between your code and a model: tokens and context as a budget, structured output and tool calls as a protocol, and prompts as versioned, tested artifacts.
Retrieval
Finding the right passages for a model to read: embeddings and approximate nearest-neighbor indexes, chunking and hybrid search with reranking, and measuring whether retrieval works.
Building blocks
The shared components every AI platform ends up with: a gateway and router in front of models, caches, queues with batch lanes, and an evaluation service.
AI infrastructure
The infrastructure under models: a training cluster, a data platform, a multi-model inference platform, and the GPU scheduler that shares them.
Agentic systems
An agent is a model choosing its own next step in a loop. This module covers what the system around that loop must guarantee: bounds, memory, durability, and control over what the agent may do.
Evaluation in production
How to measure whether a model-backed system is good enough, whether a change made it better, and whether it is still good on live traffic.
Worked AI designs
Five complete AI system designs, each argued from its invariants and sized with arithmetic: chat at scale, retrieval over private documents, a coding agent platform, a batch inference pipeline, and an evaluation platform.