Module 7 of 16
Distributed data
How replicated state agrees, how storage engines turn writes into files, how logs carry data between systems, and what it costs to commit one transaction across many machines.
The concurrency module showed what a read may return when replicas lag and what quorums do and do not promise. This module goes one layer down, into the machinery that data systems are built from. It covers the consensus protocol behind most control planes, the storage engines under most databases, the partitioned log that connects services and pipelines, and the protocols that commit one transaction across several machines.
Each lesson runs a small simulation of the mechanism: a Raft cluster through a partition, an LSM-tree counting its own write amplification, a consumer that crashes between processing and committing offsets, and a two-phase commit whose coordinator dies at the worst moment. Afterward you should be able to read a database’s or broker’s durability settings and say what a crash, a partition, or a slow disk will cost, in lost writes, blocked locks, or bytes written per byte stored.
Lessons
- 7.aConsensus and RaftHow does Raft keep one agreed log through crashes and partitions, and what does each committed write cost?24 min
- 7.bStorage enginesWhat happens to a write between the API call and the disk, and why do B-trees and LSM-trees age so differently?21 min
- 7.cLogs and stream processingHow does a partitioned log carry data between systems, and how do consumers process it effectively once despite crashes?19 min
- 7.dDistributed transactionsHow do you commit one transaction across several machines, and what happens when the coordinator fails mid-commit?18 min