Subject 05 / Course

Systems design

Invariants, failure, concurrency, scale, and data, then the same discipline applied to systems built around models: interfaces, retrieval, agents, evaluation, and AI infrastructure.

16 modules63 lessons≈ 19 h reading
Start with lesson 1.a

A system is a set of promises: this balance never goes negative, this token is used once, this request finishes or fails within two seconds. Design is the work of choosing those promises and then keeping them while requests overlap, machines fail, and load exceeds the plan. Security and reliability are the same discipline seen from two sides: both ask what still holds when something goes wrong.

The course starts from invariants and state machines, follows the ways systems fail, works through concurrency, consistency, scale, distributed data, and observability, and works four complete designs, each argued from its invariants rather than from a diagram of boxes. The second half applies the same discipline to systems built around a model, which is slow, expensive per call, non-deterministic, and wrong some of the time: model interfaces, retrieval, AI building blocks and infrastructure, agentic systems, evaluation in production, and five more worked designs. It assumes you can read Python and have built or operated a web service. It does not assume a distributed-systems background.

The path#

From a promise to a working design
flowchart LR
  I["<b>Invariants</b><br/><small>what must hold</small>"] --> F["<b>Failure</b><br/><small>partial, slow, overloaded</small>"]
  F --> C["<b>Concurrency</b><br/><small>races, leases, consistency</small>"]
  C --> P["<b>Production</b><br/><small>SLOs, change, recovery</small>"]
  P --> S["<b>Scale and data</b><br/><small>queues, shards, consensus</small>"]
  S --> W["<b>Worked designs</b><br/><small>argued end to end</small>"]
  W --> A["<b>AI systems</b><br/><small>models, retrieval, agents, evals</small>"]
  W -.->|"incidents sharpen the invariants"| I
  class I neutral
  class C memory
  class P compute
  class W good
  class S queue
  class A compute
classDef neutral fill:#f3f1ea,stroke:#5f5e55,color:#141413,stroke-width:1.5px
classDef compute fill:#f5d195,stroke:#b0420c,color:#141413,stroke-width:1.5px
classDef memory fill:#f5d195,stroke:#b0420c,color:#141413,stroke-width:1.5px
classDef queue fill:#f5d195,stroke:#b0420c,color:#141413,stroke-width:1.5px
classDef io fill:#f5d195,stroke:#b0420c,color:#141413,stroke-width:1.5px
classDef default fill:#f5d195,stroke:#b0420c,color:#141413,stroke-width:1.5px
classDef warn fill:#ef94ad,stroke:#b3154f,color:#141413,stroke-width:2.2px,stroke-dasharray:6 4
classDef good fill:#8dbb6f,stroke:#2b6a24,color:#141413,stroke-width:2.2px

How each lesson works#

  1. The idea in one minute. The claim, before the detail.
  2. A picture of the mechanism or the failure.
  3. How it really works, with the numbers and the named techniques.
  4. Code: a small Python simulation using only the standard library. Failures, clocks, and networks are simulated so every run is reproducible.
  5. Where it breaks. The assumption that, once violated, defeats the design.
  6. Remember this, Try it, and Check yourself.

What you need#

  • Python 3.10 or newer and a terminal. Every program runs offline.
  • A notebook for capacity math. Most designs fail on arithmetic before they fail on architecture.

The modules

1

Thinking in invariants

Before any boxes and arrows: what the system promises, which states it may be in, how much load it must carry, and how to choose between designs and write the choice down.

  1. 1.aWhat a design promises20 min
  2. 1.bState machines and transitions19 min
  3. 1.cBack-of-the-envelope capacity20 min
  4. 1.dTradeoffs and decision records21 min
2

How systems fail

Distributed systems rarely fail cleanly. This module covers partial failure, retries, circuit breakers, overload, and blast radius, and the controls that keep one fault from becoming an outage.

  1. 2.aPartial failure and timeouts18 min
  2. 2.bRetries and idempotency20 min
  3. 2.cCircuit breakers and deadlines21 min
  4. 2.dOverload and backpressure17 min
  5. 2.eBlast radius and cells19 min
3

Concurrency and consistency

What still holds when requests overlap, replicas lag, and processes pause: races, locks and fencing, consistency models, replication, and delivering effects exactly once.

  1. 3.aRaces and check-then-act19 min
  2. 3.bLocks, leases, and fencing17 min
  3. 3.cConsistency models18 min
  4. 3.dReplication and quorums18 min
  5. 3.eExactly-once and the outbox17 min
4

Reliability in production

How to state reliability as a number, see what a running system is doing, change it without breaking it, and get the data back when something destroys it.

  1. 4.aSLOs and error budgets21 min
  2. 4.bObservability signals20 min
  3. 4.cSafe change21 min
  4. 4.dBackup and disaster recovery21 min
5

Worked designs

Four complete designs, each argued from its invariants: a rate limiter, a credential broker, a tamper-evident audit log, and a multi-tenant inference gateway.

  1. 5.aA rate limiter21 min
  2. 5.bA credential broker23 min
  3. 5.cA tamper-evident audit log21 min
  4. 5.dA multi-tenant inference gateway20 min
6

Scale and performance

Why services get slow and how to make them fast again: queueing in pools and pipelines, caches, partitioning, load balancing against the tail, and profiling.

  1. 6.aQueueing and Little's law21 min
  2. 6.bCaching21 min
  3. 6.cPartitioning and sharding18 min
  4. 6.dLoad balancing and tail latency20 min
  5. 6.eProfiling a slow service16 min
7

Distributed data

How replicated state agrees, how storage engines turn writes into files, how logs carry data between systems, and what it costs to commit one transaction across many machines.

  1. 7.aConsensus and Raft24 min
  2. 7.bStorage engines21 min
  3. 7.cLogs and stream processing19 min
  4. 7.dDistributed transactions18 min
8

Observability

The mechanics behind telemetry at scale: metric types and cardinality, histogram error, traces and profiles, sampling and its cost, alert delays, and the signals that matter on GPU fleets.

  1. 8.aMetrics, cardinality, and histograms16 min
  2. 8.bTracing and profiling17 min
  3. 8.cSampling, cost, and alerting16 min
  4. 8.dObserving AI infrastructure18 min
9

How AI systems differ

What changes when a model sits in the request path, and how to size an AI workload in tokens, seconds, and dollars before drawing any boxes.

  1. 9.aWhat changes with a model15 min
  2. 9.bCost, latency, and quality14 min
10

Model interfaces

The contract between your code and a model: tokens and context as a budget, structured output and tool calls as a protocol, and prompts as versioned, tested artifacts.

  1. 10.aTokens, context, and cost18 min
  2. 10.bStructured output and tool calls19 min
  3. 10.cPrompts as versioned code12 min
11

Retrieval

Finding the right passages for a model to read: embeddings and approximate nearest-neighbor indexes, chunking and hybrid search with reranking, and measuring whether retrieval works.

  1. 11.aEmbeddings and ANN search18 min
  2. 11.bChunking and hybrid search17 min
  3. 11.cEvaluating retrieval15 min
12

Building blocks

The shared components every AI platform ends up with: a gateway and router in front of models, caches, queues with batch lanes, and an evaluation service.

  1. 12.aGateways and routers16 min
  2. 12.bCaches17 min
  3. 12.cQueues and batch lanes14 min
  4. 12.dEvaluation services15 min
13

AI infrastructure

The infrastructure under models: a training cluster, a data platform, a multi-model inference platform, and the GPU scheduler that shares them.

  1. 13.aA training cluster15 min
  2. 13.bA data platform15 min
  3. 13.cAn inference platform14 min
  4. 13.dGPU scheduling15 min
14

Agentic systems

An agent is a model choosing its own next step in a loop. This module covers what the system around that loop must guarantee: bounds, memory, durability, and control over what the agent may do.

  1. 14.aThe agent loop19 min
  2. 14.bMemory and planning18 min
  3. 14.cDurable execution16 min
  4. 14.dTools, permissions, and approval16 min
15

Evaluation in production

How to measure whether a model-backed system is good enough, whether a change made it better, and whether it is still good on live traffic.

  1. 15.aDesigning evals14 min
  2. 15.bModel graders14 min
  3. 15.cOnline evaluation15 min
16

Worked AI designs

Five complete AI system designs, each argued from its invariants and sized with arithmetic: chat at scale, retrieval over private documents, a coding agent platform, a batch inference pipeline, and an evaluation platform.

  1. 16.aA chat assistant at scale24 min
  2. 16.bRetrieval over private documents20 min
  3. 16.cA coding agent platform22 min
  4. 16.dA batch inference pipeline22 min
  5. 16.eAn evaluation platform19 min
← Back to the library