kafka and zookeeper

Table of Contents

Using zookeeper with kafka

  1. Used to manage a Kafka cluster, track node status, and maintain a list of topics and messages.

  2. Zookeeper manages brokers (keeps a list of them).

  3. Zookeeper helps in performing leader election for partitions.

  4. Zookeeper sends notifications to Kafka in case of changes (e.g. new topic, broker dies, broker comes up, delete topics, etc.)

    Kafka 2.x cannot work without Zookeeper.

    kaf 3.x can work without Zookeeper (KIP-500) - using Kafka Raft instead.

    kafka 4.x will not have Zookeeper.

    Kafka version 2.8.0 introduced early access to a Kafka version without Zookeeper, but it’s not ready yet for production environments.

    Zookeeper, by design, operates with an odd number of servers (1,3,5,7).

    Zookeeper has a leader (writes) the rest of the servers are followers (reads).

    Zookeeper does not store consumer offsets with Kafka > v0.10

Zookeeper Cluster (ensemble)

 Zookeeper Server 1        Zookeeper Server 2         Zookeeper Server 3
 (follower)                (Leader)                   (follower)
     |                             |                     |
     |                             |                     |
  -------------               ----------                 |
 |             |             |          |                |
 |             |             |          |                |
Broker1     Broker2        Broker3   Broker4           Broker5

Should you use Zookeeper?

With kafka brokers?

  • Yes, until kafka 4.0 is out while waiting for kafka without Zookeeper to be production ready.

With kafka clients?

  • Over time, the kafka clients and CLI have been migrated to leverage the broker as a connection endpoint instead of Zookeeper
  • Since kafka 0.10, consumers store offset in Kafka and Zookeeper and must not connect to Zookeeper as it is deprecated.
  • Since kafka 2.2, the kafka-topics.sh CLI command references kafka brokers and not Zookeeper for topic management (creation, deletion, etc.) and the Zookeeper CLI argument is deprecated.
  • All the APIs and commands that were previously leveraging Zookeeper are migrated to use Kafka instead, so that when clusters are migrated to be without Zookeeper, the change is invisible to clients.
  • Zookeeper is less secure than kafka. Therefore, Zookeeper ports should only be opened to allow traffic from kafka brokers, and not kafka clients
  • Therefore, to be a great modern-day kafka developer, never use Zookeeper as a configuration in your kafka clients, and other programs that connect to kafka.

Links to this note