Kafka - Serialization and Deserialization

  1. kafka only accepts bytes as an input from producers and sends bytes out as an output to consumers.
  2. Message serialization means transforming objects/data into bytes - a byte array.
  3. They are used on the value and the key. e.g. key object = 123 (truck id), value object (latitule + longitude of the truck)
  4. KeySerializer (Integer serializer) will transform the key into bytes.
  5. ValueSerializer (String serializer) will transform the value into bytes.
  6. It is the responsibility of the producer to convert the message into byte array (serialize) before putting it in Kafka. Similarly, it is the responsibility of the consumer to deserialize the message when it consumes a message from Kafka.
  7. kafka producers come with standard built-in serializers
    • String (includes JSON)
    • Integer, Float
    • Avro
    • Protobuf

Links to this note