Kafka - Stream Bridge

Problem statement

  1. With spring-cloud-stream, we write Producers and specify the corresponding destinations using binder properties in application.yaml file.
  2. Using this, messages are routed to the corresponding topics based on configuration.
  3. But all of this is static. Meaning, we do the configuration well in advance before deploying the application. One function/bean is mapped to one destination. It is more of a one-to-one mapping.
  4. Content-based routing
    1. What if this mapping needs to be dynamic?
    2. example scenario
      1. If a customer purchases a book on our website, we need to put a message in the digital topic if the order is for digital book and we need to put a message in the physical topic if the order is for physical book.
    3. In other words, we need to route messages to topics based on additional conditions.
    4. We cannot solve this problem simply by creating two beans - one for digital and one for physical.
      1. What if there are 100 different topics and the routing has to take all of them into account?
      2. Sometimes, we may not know what the destination topic is in advance.
      3. Routing needs to be done to dynamic destinations.

StreamBridge

  1. To send arbitrary messages to a topic
  2. To route messages based on condition
    1. odd number - odd-topic
    2. even number - even-topic
  3. Output destination is known at run-time

Sample use-case

  1. Incoming topic/event: order-event - it will have customer id, product id, order type
  2. Outgoing topic/event:
    1. Delivery topics
      1. email
      2. home-address

Links to this note