Kafka & Event Streaming
Kafka cheat sheet: topics, partitions and consumer groups in one diagram, offsets and consumer lag, at-most-once, at-least-once and exactly-once delivery, producer and durability settings, rebalancing, Kafka vs a queue, and the CLI commands.
Cheat sheet · Offered as a template
In this diagram
- Kafka & Event Streaming
- ① The model
- A topic is a named log, split into partitions
- Each record is appended and gets the next offset
- Reading removes nothing; retention does
- Order is guaranteed only inside a partition
- ② Producers, partitions, consumer groups
- topic: orders
- group: billing
- Orders API
- Checkout
- partition 0
- partition 1
- partition 2
- consumer 1
- consumer 2
- ③ Offsets and lag
- read so far · still to read
- Lag that keeps growing means consumers are slower than producers.
- ④ Delivery guarantees
- ⑤ The consumer loop
- consumer.py
- ⑥ Producer settings
- producer.properties
- ⑦ Durability
- topic config
- With acks=all a write succeeds once 2 of 3 replicas have it, so one broker can die without losing data.
- ⑧ A rebalance
- A consumer joins, leaves or stops heart-beating
- The group coordinator pauses the group
- Partitions are handed out again
- Each consumer resumes from the committed offset
- ⑨ Kafka or a queue?
- Reach for Kafka
- Many consumers read the same events
- You need to replay history
- High throughput, ordered per key
- Reach for a queue
- Per-message ack, retry and dead-letter
- Handing tasks to a pool of workers
- Low volume, simple operations
- ⑩ Choosing partitions
- More partitions means more consumers in parallel
- You can add partitions, never remove them