#storage
4 notes · ← All notes
- Build your own Kafka, 1: records on diskThe exact bytes Kafka writes for a batch of records: a 61-byte header, varint-encoded records and a CRC-32C. You write the varint encoder.
- Build your own Kafka, 2: offsets and the indexA consumer asks for offset 217. How a broker finds it in a gigabyte file without scanning: a sparse index, binary search, then a short scan. You write the lookup.
- Build your own Kafka, 3: segments and retentionAn append-only file can only grow. Splitting the log into segment files makes deleting old data cheap. You write the retention rule.
- Build your own Kafka, 4: the page cache and crash recoveryKafka does not fsync each write. What that costs when the power goes, how a broker repairs a torn log on startup, and why replication is the real answer. You write the recovery.