Cloud Design Patterns - Transactional outbox pattern

https://learn.microsoft.com/en-us/azure/architecture/databases/guide/transactional-outbox-cosmos

The problem statement

Options to solve the problem

If we can insert the record in the main table and send the message to the message platform in one single transaction, that would solve the problem. But it is not possible.

Instead, what we can do is, insert the record in the main table and insert another record in another table called “transactional outbox table” and do these two insertions in a single transaction.

Later, we can just read the “transactional outbox table”, and send events for the records in this table to the messaging platform.

Outbox table

Design with Transactional Outbox table

Considering the possible issues with this design:

1. If the communication with the message broker fails, the application will just read the row from the table later.
2. If the communication with the message broker is successful, but the application fails to update/delete the rows from the “transactional outbox table” due to a crash, there will be duplicate events in the message broker.

Challenges with event driven architecture

1. Things will be eventually consistent
2. Duplicate events are possible

We accept these challenges and design our systems as best as we can. We have to account for these two possibilities in our designs.

Goal

Alternatives

Option 1 - Catch and handle “duplicate event exceptions” before processing

1. Don’t use a special table.
2. From the service layer, throw EventAlreadyProcessedException.
3. Before processing the process() method, see if EventAlreadyProcessedException is thrown.
   1. If it is thrown, it means that the message was already processed.
   2. Just build a dto and acknowledge the message.

Option 2 - Database triggers

1. Using a database trigger.
2. When a row is inserted into the actual table, like inventory table, etc. create a row in a duplicate table.
3. This duplicate table will act as the Outbox table.
4. And use the rows from this duplicate table for messaging.
5. With this approach, we can eliminate internal “listeners” in the messaging layer completely.

Option 3 - Change Data Capture (CDC)

https://debezium.io/