Kafka - How are brokers, topics and partitions related
How are brokers, topics and partitions related?
-
Each Broker (or Node or Server) in a kafka cluster can/will have more than one topic assigned to them.
-
A Broker can be the primary Node for one type of events (order events) but at the same time, it can be a back-up Node for another type of events (taxi-gps).
-
Essentially, the data related to events is distributed across many, many Nodes in the cluster. If any one Node dies for whatever reason, there will always be back-up for the data that was previously held in that Node.
-
Example Scenario:
- Three brokers Broker101, Broker102 and Broker103
- TopicA with 3 partitions and TopicB with 2 partitions
-
Data is distributed.
This distribution lets kafka scale horizontally. -
Broker103 does not have any TopicB data.
If we call the topics Topic A and Topic B, and the partitions for each of the topics to be TAP0, TAP1, TAP2, TBP0 and TBP2, here are the possible combinations that we can have on a given node.
| Description of the Broker | Topic A | Topic B |
|---|---|---|
| Leader for Partition 0 of Topic A | P0 | P0 |
| P0 | P1 | |
| P0 | P0, P1 | |
| P1 | P0 | |
| P1 | P1 | |
| P1 | P0, P1 | |
| Leader for Partition 0 of Topic B | P2 | P0 |
| P2 | P1 | |
| P2 | P0, P1 | |
| P0, P1 | P0 | |
| Leader for Partition 1 of Topic A | P1, P2 | P0 |
| P2, P0 | P0 | |
| P0, P1, P3 | P0 | |
| Leader for Partition 1 of Topic B | P0, P1 | P1 |
| P1, P2 | P1 | |
| P2, P0 | P1 | |
| Leader for Partition 3 of Topic A | P0, P1, P3 | P1 |
One single broker will not be the leader for all the partitions of a topic.