Using Change Data Capture and Polling Publisher in the Transactional Outbox Pattern

Event driven architecture has been used in scalable, resilient systems for a long time, providing significant advantages such as decoupling, asynchronous processing, and improved fault tolerance. However, delivering events reliably can be challenging, especially when services have separate databases. In order to address this, Transactional Outbox Pattern has become a widely adopted approach in event driven systems.

What Does the Transactional Outbox Pattern Solve?

Let’s start with a code snippet to discuss what problems can occur when the transactional outbox pattern is not used.

    @Transactional
    public void handleOrderCreatedEvent(OrderCreatedEvent orderCreatedEvent) {
        // Save the event to the database
        orderRepository.save(orderCreatedEvent);

        // Send the event to the Kafka
        kafkaTemplate.send("order-topic", orderCreatedEvent);
    }

At first glance, this code might not seem problematic because it saves to the database and then sends the event to Kafka. However, imagine that Kafka throws an exception after the database save operation. Due to the @Transactional annotation, the database operation is rolled back. This can lead to data loss.

The scenario where the event cannot be sent to Kafka due to unknown error

Another potential issue occurs in case of a network partition. Kafka might receive the event but fail to return an acknowledgment to the producer. As a result, the producer assumes that there is a failure and triggers a rollback even though Kafka might have successfully received the event. In the end, the event is delivered to the consumer, but the producer loses the event due to the rollback. Lastly, if the producer has retry logic for Kafka, then dual write problem is more likely to occur (Idempotency is the rescue!). As a result, it can be said that saving events to database and sending them to Kafka cannot be achieved in a transactional way.

The scenario where the acknowledgement cannot be sent to the producer due to network partition

Due to scenarios like above, the Transactional Outbox Pattern is widely used in most event driven systems to ensure that events are reliably delivered to the message broker. There are two frequently used Transactional Outbox Pattern designs which are Change Data Capture (CDC) and Polling Publisher.

Change Data Capture

CDC is a technique used to identify and capture changes made to data in a database and it enables near-real-time data synchronization across systems. Instead of scanning the entire database for updates, CDC captures only the modifications such as inserts, updates, and deletes by reading transaction log (Write-Ahead Log). This approach is highly efficient for distributed systems that need to propagate data changes to other systems.

In the context of Transactional Outbox Pattern, CDC allows applications to detect changes to specific tables and it ensures that captured changes are reliably published as events to messaging brokers such as Kafka. Debezium is one of the CDC tools that is widely used in distributed systems and the implementation of the outbox pattern with Debezium is demonstrated as below;

Transactional Outbox Pattern with Debezium

Some advantages of using CDC in Transactional Outbox Pattern;

• Near Real-Time Data Propagation: Debezium captures changes in near real-time. This enables low-latency communication between services.
• Decoupling of Systems: It ensures that changes made in the database are captured and published as events without integrating anything in producer (such as Kafka integration).
• Automatic recovery: If Debezium crashes, it can resume from the point where it left by reading the database logs.
• Easy integrations: Debezium integrates very well with Kafka by providing connectors.

There are some disadvantages as well;

• Database Support: The database that is in use must support CDC.
• Setup of Debezium: Debezium needs to be configured, monitored, and managed separately.

Polling Publisher

This pattern is another way to implement the Transactional Outbox Pattern. After an event is stored to the outbox table, polling publisher service periodically retrieves unprocessed events from the outbox table and sends them to message broker. The architecture of this pattern is illustrated as below;

Transactional Outbox Pattern with Polling Publisher

Some advantages of this pattern;

• Database Independent: Unlike CDC tools, this pattern can work with any relational database.
• Easy to implement: It is just a service that needs to be implemented with a couple of logic.

On the other hand there are also disadvantages such as;

• Database Load: Since this pattern scans the outbox table at fixed intervals, it can a create heavy load on the database if the interval is set very short.
• Latency: As this pattern runs at fixed intervals, it cannot achieve near real-time updates compared to CDC.

To sum up, both approaches have their own advantages and disadvantages. If real-time updates are required, CDC might be a better solution. On the other hand, polling publisher pattern could be a good choice for systems that do not need real-time updates as it is easier to implement. Lastly, it is important to mention that in both approaches, dual write problem may still occur, but it can be discussed in another article(For those who are curious about it, please read about idempotent producer and consumer).