Kafka - Delivery semantics

Reading material

1. https://docs.confluent.io/kafka/design/delivery-semantics.html
2. https://learn.conduktor.io/kafka/delivery-semantics-for-kafka-consumers/
3. https://www.baeldung.com/kafka-message-delivery-semantics

Delivery semantics for consumers

1. By default, Java Consumers will automatically commit offsets (At least once).
2. If you choose to commit manually, there are 3 delivery semantics.
   • At least once (usually preferred)
     • Offsets are committed after the message is processed
     • If the processing goes wrong, the message will be read again
     • This can result in duplicate processing of messages. Make sure your processing is idempotent (i.e. processing the messages again will not impact your systems)
   • At most once
     • Offsets are committed as soon as messages are received.
     • If the processing goes wrong, some messageswill be lost (they will not be read again)
   • Exactly once
     • For kafka => kafka workflows: use the Transactional API (easy with Kafka Streams API)
     • For kafka => External System workflows: use an idempotent consumer