Streaming
What Is Streaming?
Streaming is the continuous transfer of data where data can be processed as it comes in. Streaming platforms allow for real-time data processing, allowing immediate insights and actions based on the incoming data.
A streaming broker is different from a message queue for many reasons. For starters, messages are organized into log files or topics. One or more consumers can subscribe to a log file or topic to receive all messages that come through that stream. With proper setup, a streaming broker will deliver the same message to every subscriber, in a specific order. This is often described as a publish-subscribe pattern.
Active subscribers will always get the message, but with this type of message broker, new subscribers can access the logs file and read messages from any point in time. This has its own set of benefits for many use cases.
Pros and Cons of Stream Processing
A streaming broker acts as a distributed data store, which means you can use it as a single source of truth when working with data-centric environments that require real-time interactions. Let’s say you need to expose a given message in many formats (such as graph, search, and so on). A message queue would make that impossible because a message can only be received once.
A streaming broker makes taking one message and producing multiple actions easy with the help of polyglot persistence. In other words, a streaming broker can deliver the same message to multiple consumers, with each consumer set up to do a different thing with that message. This is known as reactive programming, whereas a message queue is designed for imperative programming.
- Streaming brokers can deliver the same message to many consumers without the need for replication.
- Consumers can be set up to do different things with the same message.
- Streaming brokers always deliver in the same order messages are queued.
Products
- Apache Kafka
- Amazon Kinesis
- Redpanda