Kafka - Fan in and Fan out

Table of Contents

Fan out

  1. Scenarios where the Producer has to produce messages to multiple topics (usually done in parallel).
  2. One to many
  3. Example scenarios:
    1. SMS notification for an order - use sms topic
    2. email notification for an order - use email topic
  4. Why don’t the Producer write to one topic and let the Consumers use different consumer groups.
    1. sms consumer application belongs to one consumer group.
    2. email consumer application belongs to another consumer group.
    3. Both of these applications can subscribe to the same topic and read messages from the same topic.
    4. It is possible to do it that way. But this is not a good approach.
      1. Each application has to consume all the messages from the topic and then use a filter to see if each message is applicable to it or not. It is not a very efficient approach.
      2. We cannot control features like message retention period for sms and email messages in the topic if both of them use the same topic.

Fan in

  1. One to many
  2. If multiple Producers are producing messages into multiple formats, in different formats, and if our application needs to consume all those messages, then we need to use this pattern.
  3. Example scenario: Ride Share service
    1. Messages about waiting passengers will be coming in from one topic.
    2. Messages about available drivers will be coming in from another topic.
    3. Our application has to pick up the first message from the “passengers-topic” and pair it with the first message from “available-drivers” topic and zip them and send notifications to both of them.

Links to this note