Build your own Kafka
Every chapter starts with something that breaks. You write the one function that fixes it, the tests go green, and the demo next to it switches to your code. Then you break it on purpose and compare with the class in Kafka that does the same job.
Chapters
- Chapter 0 Build your own Kafka, 0: why a log
Before writing any code: why Kafka stores messages in an append-only log instead of a queue, and what that one decision buys.
- Chapter 1 Build your own Kafka, 1: records on disk
The exact bytes Kafka writes for a batch of records: a 61-byte header, varint-encoded records and a CRC-32C. You write the varint encoder.
- Chapter 2 Build your own Kafka, 2: offsets and the index
A consumer asks for offset 217. How a broker finds it in a gigabyte file without scanning: a sparse index, binary search, then a short scan. You write the lookup.
- Chapter 3 Build your own Kafka, 3: segments and retention
An append-only file can only grow. Splitting the log into segment files makes deleting old data cheap. You write the retention rule.
- Chapter 4 Build your own Kafka, 4: the page cache and crash recovery
Kafka does not fsync each write. What that costs when the power goes, how a broker repairs a torn log on startup, and why replication is the real answer. You write the recovery.
Coming next: compaction, partitions and keys, idempotent producers, replication and the high watermark, ISR and acks, leader epochs, KRaft, consumer groups, committed offsets and transactions.