Subject 05 / Course

Systems design

Invariants, failure modes, concurrency, and the tradeoffs behind systems that keep their promises.

5 modules22 lessons≈ 7 h reading
Start with lesson 1.a

A system is a set of promises: this balance never goes negative, this token is used once, this request finishes or fails within two seconds. Design is the work of choosing those promises and then keeping them while requests overlap, machines fail, and load exceeds the plan. Security and reliability are the same discipline seen from two sides: both ask what still holds when something goes wrong.

The course starts from invariants and state machines, follows the ways systems fail, works through concurrency and consistency, and ends with four complete designs, each one argued from its invariants rather than from a diagram of boxes. It assumes you can read Python and have built or operated a web service. It does not assume a distributed-systems background.

The path#

From a promise to a working design
flowchart LR
  I["<b>Invariants</b><br/><small>what must hold</small>"] --> F["<b>Failure</b><br/><small>partial, slow, overloaded</small>"]
  F --> C["<b>Concurrency</b><br/><small>races, leases, consistency</small>"]
  C --> P["<b>Production</b><br/><small>SLOs, change, recovery</small>"]
  P --> W["<b>Worked designs</b><br/><small>argued end to end</small>"]
  W -.->|"incidents sharpen the invariants"| I
  class I neutral
  class F warn
  class C memory
  class P compute
  class W good
classDef neutral fill:#faf9f5,stroke:#5f5e55,color:#141413,stroke-width:1.5px
classDef compute fill:#f9e8cb,stroke:#b0420c,color:#141413,stroke-width:1.5px
classDef memory fill:#f9e8cb,stroke:#b0420c,color:#141413,stroke-width:1.5px
classDef queue fill:#f9e8cb,stroke:#b0420c,color:#141413,stroke-width:1.5px
classDef io fill:#f9e8cb,stroke:#b0420c,color:#141413,stroke-width:1.5px
classDef default fill:#f9e8cb,stroke:#b0420c,color:#141413,stroke-width:1.5px
classDef warn fill:#f9e2e9,stroke:#b3154f,color:#141413,stroke-width:2px,stroke-dasharray:5 3
classDef good fill:#e2efd8,stroke:#2b6a24,color:#141413,stroke-width:2px

How each lesson works#

  1. The idea in one minute. The claim, before the detail.
  2. A picture of the mechanism or the failure.
  3. How it really works, with the numbers and the named techniques.
  4. Code: a small Python simulation using only the standard library. Failures, clocks, and networks are simulated so every run is reproducible.
  5. Where it breaks. The assumption that, once violated, defeats the design.
  6. Remember this, Try it, and Check yourself.

What you need#

  • Python 3.10 or newer and a terminal. Every program runs offline.
  • A notebook for capacity math. Most designs fail on arithmetic before they fail on architecture.

The modules

1

Thinking in invariants

Before any boxes and arrows: what the system promises, which states it may be in, how much load it must carry, and how to choose between designs and write the choice down.

  1. 1.aWhat a design promises20 min
  2. 1.bState machines and transitions19 min
  3. 1.cBack-of-the-envelope capacity20 min
  4. 1.dTradeoffs and decision records21 min
2

How systems fail

Distributed systems rarely fail cleanly. This module covers partial failure, retries, circuit breakers, overload, and blast radius, and the controls that keep one fault from becoming an outage.

  1. 2.aPartial failure and timeouts18 min
  2. 2.bRetries and idempotency20 min
  3. 2.cCircuit breakers and deadlines20 min
  4. 2.dOverload and backpressure17 min
  5. 2.eBlast radius and cells19 min
3

Concurrency and consistency

What still holds when requests overlap, replicas lag, and processes pause: races, locks and fencing, consistency models, replication, and delivering effects exactly once.

  1. 3.aRaces and check-then-act19 min
  2. 3.bLocks, leases, and fencing17 min
  3. 3.cConsistency models18 min
  4. 3.dReplication and quorums18 min
  5. 3.eExactly-once and the outbox17 min
4

Reliability in production

How to state reliability as a number, see what a running system is doing, change it without breaking it, and get the data back when something destroys it.

  1. 4.aSLOs and error budgets21 min
  2. 4.bObservability signals20 min
  3. 4.cSafe change21 min
  4. 4.dBackup and disaster recovery21 min
5

Worked designs

Four complete designs, each argued from its invariants: a rate limiter, a credential broker, a tamper-evident audit log, and a multi-tenant inference gateway.

  1. 5.aA rate limiter21 min
  2. 5.bA credential broker23 min
  3. 5.cA tamper-evident audit log21 min
  4. 5.dA multi-tenant inference gateway20 min
← Back to the library