Kafka - Producers

Kafka producer

1. An application (a piece of code) you write to get data to Kafka topics (which are made of partitions).
2. Producers know (in advance) to which partition they should write data to (and which kafka broker has the partition).
3. In case of kafka broker failures, Producers will know how to automatically recover.

Push or Pull architecture?

1. Producers use the push model and consumers use the pull model.

Producer acknowledgements (ack)

https://kafka.apache.org/22/generated/producer_config.html

Valid values: [all, -1, 0, 1]

Default value: -1 (to allow maximum throughput)

Description:

1. acks=0, If set to zero then the producer will not wait for any acknowledgment from the server at all. The record will be immediately added to the socket buffer and considered sent. No guarantee can be made that the server has received the record in this case, and the retries configuration will not take effect (as the client won’t generally know of any failures). The offset given back for each record will always be set to -1.

2. acks=1 This will mean the leader will write the record to its local log but will respond without awaiting full acknowledgement from all followers. In this case should the leader fail immediately after acknowledging the record but before the followers have replicated it then the record will be lost.

3. acks=all This means the leader will wait for the full set of in-sync replicas to acknowledge the record. This guarantees that the record will not be lost as long as at least one in-sync replica remains alive. This is the strongest available guarantee. This is equivalent to the acks=-1 setting.

   1. Kafka Producer Acks Deep Dive - https://www.conduktor.io/kafka/kafka-producer-acks-deep-dive
   2. Producers send data into kafka brokers.
   3. Producers can choose to receive acknowledgements of data writes:
      • acks=0: Producer will not wait for acknowledgement (possible data loss)
      • acks=1: Producer will wait for leader acknowledgement (limited data loss)
      • acks=all: Leader + replicas acknowledgement (no data loss)

   What does an acknowledgement actually mean? The default behavior is: the message was written to the Leader broker + written to the replication brokers.

   If the producer is waiting for an acknowledgement, and it doesn’t receive one, what happens then?

   The client library that we use to work with kafka (e.g. reactor-kafka), will retry.

Producer Configuration

https://kafka.apache.org/22/generated/producer_config.html

min.insync.replicas

https://kafka.apache.org/documentation/#brokerconfigs_min.insync.replicas

When a producer sets acks to “all” (or “-1”), min.insync.replicas specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful. If this minimum cannot be met, then the producer will raise an exception (either NotEnoughReplicas or NotEnoughReplicasAfterAppend).

When used together, min.insync.replicas and acks allow you to enforce greater durability guarantees. A typical scenario would be to create a topic with a replication factor of 3, set min.insync.replicas to 2, and produce with acks of “all”. This will ensure that the producer raises an exception if a majority of replicas do not receive a write.

max_inflight

 /**
 * Returns the maximum number of in-flight records that are fetched
 * from the outbound record publisher while acknowledgements are pending.
 * @return maximum number of in-flight records
 */
@NonNull
int maxInFlight();

/**
 * Configures the maximum number of in-flight records that are fetched
 * from the outbound record publisher while acknowledgements are pending.
 * This limit must be configured along with {@link ProducerConfig#BUFFER_MEMORY_CONFIG}
 * to control memory usage and to avoid blocking the reactive pipeline.
 * @return sender options with new in-flight limit
 */
@NonNull
SenderOptions<K, V> maxInFlight(@NonNull int maxInFlight);

Message compression

https://www.confluent.io/blog/apache-kafka-message-compression/