Module 2 of 5
How systems fail
Distributed systems rarely fail cleanly. This module covers partial failure, retries, circuit breakers, overload, and blast radius, and the controls that keep one fault from becoming an outage.
On one machine, a call either returns or the process crashes. Across a network a call can succeed, fail, or simply never come back, and the caller cannot tell which. Most outages are not one broken component. They are healthy components reacting badly to a slow one: retrying too hard, queueing work nobody is waiting for, or sharing a fate they did not need to share.
The five lessons follow a failure as it spreads: partial failure and the timeouts that detect it, retries and the idempotency that makes them safe, circuit breakers and deadlines that stop waiting on a lost cause, overload and the backpressure that keeps goodput up, and the cells and shuffle shards that keep a bad tenant or a bad deploy contained. Each lesson simulates the failure in a short Python program so you can see the numbers move. By the end you should be able to look at a call graph and say what happens to every caller when one dependency gets slow, and which control bounds the damage.
Lessons
- 2.aPartial failure and timeoutsWhat does a timed-out call tell you, and how long should you wait before giving up?18 min
- 2.bRetries and idempotencyWhen is it safe to retry, and how do you keep retries from duplicating work or flattening a struggling service?20 min
- 2.cCircuit breakers and deadlinesHow do you stop waiting on a dependency that is failing, and stop doing work nobody is waiting for?20 min
- 2.dOverload and backpressureWhat happens when load exceeds capacity, and how do you keep doing useful work instead of collapsing?17 min
- 2.eBlast radius and cellsWhen something breaks, how do you make sure it takes down a small part of the system instead of all of it?19 min