tracker
→ clicks · 15/s · acks=1
No keys (sticky)
sent 0 · acked 0 · failed 0
Kafka, live
Three brokers, two topics, two consumer groups, all running right here. Take a guided scenario, or break things yourself: stop a broker, cut its power, add a consumer, drag the timeline back to see it again, or type kafka-topics --describe in the terminal.
Unclean leader election
Step 1 of 4
One replica in sync
tracker writes with acks=1. Both followers slow down and drop out of the ISR (in-sync replicas)The replicas that have kept up with the leader within replica.lag.time.max.ms. Only they may become leader in a clean election.; only the leader is left.
Based on Jack Vanlightly, "How to Lose Messages on a Kafka Cluster"
How to read this picture
Brokers
- Power light. Green: running. Red: stopped.
- Activity light. Blinks while records go in or out.
- The KRaft controller. It decides which replica leads each partition.
- In and Out: network bytes per second. Disk: bytes in its log files. Cache: bytes written but not yet on disk.
- Dashed border: a slow broker. Its followers fall behind.
- Grey stripes: a stopped broker.
- Leader replica: takes writes and serves reads
- Follower replica: copies the leader
- Out of the ISR: too far behind to count
- Number on the right: the log end offset, the next offset this replica will write.
A partition’s log
- A record; the colour is its key
- No key
- Striped: only in the page cache, lost if the power goes
- Faded: above the high watermark, not yet on every ISR replica, so consumers can’t read it
- Red inner border: a tombstone (null value) that deletes its key in a compacted topic
- Grey hatching: deleted by retention or compaction
- Dotted: an offset this replica doesn’t have yet
- Red outline: differs from the leader at the same offset
- High watermark: consumers read only below this line
- A follower’s own idea of the high watermark, which lags the leader’s
- Blue mark (in Inspect): where a consumer group will read next
- Gold frame: the partition you picked. The Inspect panel below shows it.
Producers and consumers
- Envelope: a producer. acks says how many replicas must have a record before it counts as written.
- Dashed producer: records are waiting in it because no leader can be reached.
- Face: one consumer in a group. Lag is how many records it hasn’t read yet.
- Group: its state (Stable, rebalancing…), assignor and generation, which goes up at every rebalance.
What moves
- Solid square: a record on its way from a producer to the leader (colour = key)
- Hollow square: a follower copying records from the leader
- Round dot: records fetched by a consumer
- Crossed square: a request that failed
Producers
Cluster
Broker 1
ControllerRunning
In0 B/s
Out0 B/s
Disk0 B
Cache0 B
Broker 2
Running
In0 B/s
Out0 B/s
Disk0 B
Cache0 B
Broker 3
Running
In0 B/s
Out0 B/s
Disk0 B
Cache0 B
Consumer groups
No consumer groups yet.
clicks · partition 0
- Leader
- 2
- Leader epoch
- 0
- Replicas
- 2, 3, 1
- In-sync replicas
- 2, 3, 1
- Log start offset
- 0
- High watermark
- 0
- Log end offset
- 0
Click a record to follow its journey.
Segment files on broker 2 Leader
00000000000000000000.log0 records0 B.index .timeindex
Segment files on broker 3 ISR
own HW 0
00000000000000000000.log0 records0 B.index .timeindex
Segment files on broker 1 ISR
own HW 0
00000000000000000000.log0 records0 B.index .timeindex
HW: consumers read below this line · LEO: next offset to write · only in the page cache · differs from the leader at the same offset
NOTES
Kafka notes
The concepts behind what you see above. They also live under Notes, tagged #kafka.
- Build your own Kafka, 0: why a logBefore writing any code: why Kafka stores messages in an append-only log instead of a queue, and what that one decision buys.
- Build your own Kafka, 1: records on diskThe exact bytes Kafka writes for a batch of records: a 61-byte header, varint-encoded records and a CRC-32C. You write the varint encoder.
- Build your own Kafka, 2: offsets and the indexA consumer asks for offset 217. How a broker finds it in a gigabyte file without scanning: a sparse index, binary search, then a short scan. You write the lookup.
- Build your own Kafka, 3: segments and retentionAn append-only file can only grow. Splitting the log into segment files makes deleting old data cheap. You write the retention rule.
- Build your own Kafka, 4: the page cache and crash recoveryKafka does not fsync each write. What that costs when the power goes, how a broker repairs a torn log on startup, and why replication is the real answer. You write the recovery.
- Kafka: consumer groups, rebalances and lagHow a group splits partitions between its members, what happens when someone joins, leaves or crashes, where committed offsets live, and how to read lag.
- Kafka: topics, partitions, offsets and the logA topic is a set of append-only logs. How records get an offset, which partition a key lands on, and what the files on a broker’s disk look like.
- Logging in to Kafka and MySQL from a laptop with a client certificateA private CA, a client certificate, an SSH tunnel, and DataGrip: the command to run every time, the one-time setup, and every question that came up on the way.
- Kafka: replication, the ISR and the high watermarkLeaders and followers, which replicas count as in sync, what acks=all really waits for, and how Kafka avoids (or allows) losing writes when a broker dies.