#kafka
9 notes · ← All notes
- Build your own Kafka, 0: why a logBefore writing any code: why Kafka stores messages in an append-only log instead of a queue, and what that one decision buys.
- Build your own Kafka, 1: records on diskThe exact bytes Kafka writes for a batch of records: a 61-byte header, varint-encoded records and a CRC-32C. You write the varint encoder.
- Build your own Kafka, 2: offsets and the indexA consumer asks for offset 217. How a broker finds it in a gigabyte file without scanning: a sparse index, binary search, then a short scan. You write the lookup.
- Build your own Kafka, 3: segments and retentionAn append-only file can only grow. Splitting the log into segment files makes deleting old data cheap. You write the retention rule.
- Build your own Kafka, 4: the page cache and crash recoveryKafka does not fsync each write. What that costs when the power goes, how a broker repairs a torn log on startup, and why replication is the real answer. You write the recovery.
- Kafka: consumer groups, rebalances and lagHow a group splits partitions between its members, what happens when someone joins, leaves or crashes, where committed offsets live, and how to read lag.
- Kafka: topics, partitions, offsets and the logA topic is a set of append-only logs. How records get an offset, which partition a key lands on, and what the files on a broker’s disk look like.
- Logging in to Kafka and MySQL from a laptop with a client certificateA private CA, a client certificate, an SSH tunnel, and DataGrip: the command to run every time, the one-time setup, and every question that came up on the way.
- Kafka: replication, the ISR and the high watermarkLeaders and followers, which replicas count as in sync, what acks=all really waits for, and how Kafka avoids (or allows) losing writes when a broker dies.