Modern distributed systems often need to propagate database changes to other systems almost immediately.

For example, when an order is created or updated, we may need to:

• update a search index
• invalidate a Redis cache
• send data to analytics
• notify another microservice
• publish an event to Kafka

A simple implementation might look like this:

Application
     ↓
Update Database
     ↓
Publish Kafka Event

At first glance, this looks perfectly reasonable.

But what happens if the database update succeeds and the Kafka publish fails?

Database Update   ✅
Kafka Publish     ❌

The database now contains the new business state, but downstream services never receive the corresponding event.

This is the classic dual-write consistency problem.

In this article, we will understand how Change Data Capture (CDC) and the Transactional Outbox Pattern work together to solve this problem, and how technologies such as PostgreSQL WAL, MySQL Binlog, Oracle Redo Logs, Debezium, AWS DMS, Apache Kafka, and Amazon MSK fit into the architecture.

What Is Change Data Capture?

Change Data Capture, commonly called CDC, is a technique used to detect changes occurring inside a database and propagate those changes to another system.

Typical database changes include:

INSERT
UPDATE
DELETE

A naive solution would be to continuously poll the database:

SELECT *
FROM orders
WHERE updated_at > ?;

But CDC generally takes a different approach.

Instead of repeatedly querying application tables, CDC reads the database’s transaction log.

The basic architecture becomes:

Application
    ↓
Database
    ↓
Transaction Log
    ↓
CDC Connector
    ↓
Kafka / Target System

This allows database changes to be captured in near real time without continuously scanning large application tables.

---

Database Transaction Logs

Different database engines use different transaction-log technologies.

Database	Transaction Log
PostgreSQL	WAL — Write-Ahead Log
MySQL	Binlog — Binary Log
Oracle	Redo Logs / Archived Redo Logs

Suppose we execute:

UPDATE orders
SET status = 'PAID'
WHERE order_id = 'ORD-1001';

In PostgreSQL, the corresponding database change is recorded in the WAL.

Conceptually:

Order Updated
     ↓
PostgreSQL WAL
     ↓
CDC Connector
     ↓
Kafka Event

---

Why Does CDC Read Transaction Logs Instead of Tables?

Imagine a database containing hundreds of millions of rows.

Continuously running queries such as:

SELECT *
FROM orders
WHERE updated_at > ?;

can create additional database load.

Transaction logs already contain information describing what changed.

That gives CDC several advantages:

• lower application-table query overhead
• near real-time change detection
• reliable position tracking
• INSERT, UPDATE and DELETE capture
• better scalability than repeated table polling

---

CDC with Debezium

Debezium is a popular open-source CDC platform commonly used with Kafka.

A typical architecture looks like this:

PostgreSQL / MySQL
        ↓
   WAL / Binlog
        ↓
    Debezium
        ↓
  Kafka Connect
        ↓
 Apache Kafka
        ↓
   Consumers

Suppose the orders table changes from:

status = CREATED

to:

status = PAID

Debezium can detect that transaction-log change and publish an event representing the update.

---

CDC on AWS with AWS DMS

Another important CDC technology is AWS Database Migration Service — AWS DMS.

AWS DMS supports three common modes:

• Full Load
• Full Load + CDC
• CDC Only

A typical migration architecture could be:

On-Prem Oracle
      ↓
   AWS DMS
      ↓
Aurora PostgreSQL

DMS can first copy the historical data and then continue replicating new changes.

---

Full Load + CDC

Suppose we are migrating:

On-Prem Oracle
       ↓
Aurora PostgreSQL

The target database is initially empty.

We therefore need two phases.

Step 1 — Full Load

Copy the existing historical records.

Step 2 — CDC

Capture new changes occurring in the source while or after the historical data is being migrated.

Conceptually:

Source Database
      │
      ├── Existing Data ──→ Full Load
      │
      └── New Changes ────→ CDC
                               ↓
                         Target Database

This reduces migration downtime because the source database can continue accepting transactions while the migration progresses.

---

When Do We Use CDC Only?

CDC-only mode is useful when the target already contains the required baseline data or when the purpose is continuous event streaming rather than historical migration.

Example:

PostgreSQL
    ↓
   CDC
    ↓
Amazon MSK

Here, we may only be interested in future database changes.

A useful mental rule is:

New Target
→ Full Load + CDC

Existing Synchronized Target
or Event-Streaming Use Case
→ CDC Only

---

AWS DMS vs Debezium

A common architecture question is:

If we use AWS DMS, do we also need Debezium?

Usually, no.

Both can perform CDC, but they are commonly selected for different architectural goals.

Requirement	Common Choice
Database migration	AWS DMS
Full load + ongoing replication	AWS DMS
Kafka-centric CDC platform	Debezium
Continuous event streaming	Debezium + Kafka
AWS-managed migration	AWS DMS

A simple mental model:

Migration
On-Prem Oracle
      ↓
   AWS DMS
      ↓
Aurora PostgreSQL

versus:

Event Streaming
PostgreSQL
     ↓
 Debezium
     ↓
Amazon MSK
     ↓
Microservices

You normally do not need both DMS and Debezium in the same source-to-target CDC pipeline simply because both support CDC.

---

CDC with Amazon MSK

For AWS-based Kafka architectures, Amazon MSK can act as the streaming platform.

PostgreSQL
    ↓
   WAL
    ↓
Debezium
    ↓
Amazon MSK
    ↓
Consumer Services

Possible consumers include:

• Search Service
• Cache Invalidation Service
• Analytics Service
• Notification Service
• Data Lake Pipeline

---

The Limitation of Plain CDC

CDC is powerful, but capturing raw business-table changes introduces another design problem.

Suppose placing an order modifies several tables:

orders
order_items
inventory
payment_status

CDC may observe changes such as:

orders.status changed
inventory.quantity changed
payment_status inserted

But downstream services may not care about those low-level database mutations.

They may actually want a meaningful business event:

ORDER_CREATED

or:

ORDER_PAID

This is where the Transactional Outbox Pattern becomes extremely useful.

---

What Is the Transactional Outbox Pattern?

Instead of asking the application to independently:

Write Database
+
Publish Kafka Event

the application writes both the business state and the business event into the same database transaction.

Application Transaction
        │
        ├── Update Business Table
        │
        └── Insert Outbox Event

Both operations commit together.

If the transaction succeeds:

Business record ✅
Outbox event    ✅

If the transaction fails:

Business record ❌
Outbox event    ❌

This removes the dangerous condition where the database state changes but the related event is lost.

---

Business Table vs Outbox Table

This is one of the most important concepts to understand.

Business Table

The business table represents the current state of the business entity.

Example:

orders

order_id	customer_id	amount	status	created_at
ORD-1001	CUST-501	2499.00	CREATED	10:15:20
ORD-1002	CUST-205	1299.00	PAID	10:18:42

This table answers:

What is the current state of the order?

Outbox Table

The Outbox table represents:

What business event happened?

Example:

outbox_events

event_id	aggregate_id	event_type	payload	created_at
EVT-9001	ORD-1001	ORDER_CREATED	JSON	10:15:20
EVT-9002	ORD-1002	ORDER_PAID	JSON	10:18:42

A payload might look like:

{
  "eventId": "EVT-9001",
  "eventType": "ORDER_CREATED",
  "orderId": "ORD-1001",
  "customerId": "CUST-501",
  "amount": 2499.00,
  "status": "CREATED",
  "eventTime": "2026-08-30T10:15:20"
}

The important distinction is:

Business Table
→ Current business state

Outbox Table
→ Business event that occurred

---

Atomic Transaction Example

Conceptually, the application could execute:

BEGIN;

INSERT INTO orders
(
    order_id,
    customer_id,
    amount,
    status
)
VALUES
(
    'ORD-1001',
    'CUST-501',
    2499,
    'CREATED'
);

INSERT INTO outbox_events
(
    event_id,
    aggregate_id,
    event_type,
    payload
)
VALUES
(
    'EVT-9001',
    'ORD-1001',
    'ORDER_CREATED',
    '{ ... JSON payload ... }'
);

COMMIT;

The important concept is:

Business Write
      +
Outbox Write
      =
Same Database Transaction

---

Does CDC Read the Outbox Table Directly?

This is a common misunderstanding.

In a log-based CDC architecture, the CDC connector normally reads:

WAL / Binlog / Redo

It does not need to continuously execute:

SELECT * FROM outbox_events;

The actual flow is:

Application
     ↓
Orders Table + Outbox Table
     ↓
Database Transaction
     ↓
WAL / Binlog / Redo
     ↓
CDC Connector
     ↓
Kafka

The Outbox insert itself becomes part of the database transaction log.

CDC detects that change and publishes the corresponding event downstream.

---

Why Outbox Is Better Than Direct Kafka Publishing

Without Outbox:

Application
    │
    ├── Update DB      ✅
    │
    └── Publish Kafka  ❌

Result:

Database says:
ORDER_CREATED

Kafka says:
No event

The systems are inconsistent.

With Outbox:

Application
     ↓
Single Database Transaction
     │
     ├── Business Update
     └── Outbox Event
              ↓
             CDC
              ↓
            Kafka

The application no longer needs to coordinate a distributed transaction between the database and Kafka.

This is conceptually similar to the dual-write problem in caching: independently updating a database and another system can create inconsistency when only one operation succeeds.

You can internally link your earlier article here:

Read next: Distributed Caching with Redis — Cache-Aside, TTL, Hot Keys & Redis Cluster

---

Kafka Ordering with CDC and Outbox

Kafka guarantees ordering within a partition, not across an entire topic.

Suppose an order produces:

ORDER_CREATED
ORDER_PAID
ORDER_SHIPPED

For all events belonging to:

ORD-1001

use:

Kafka Key = ORD-1001

Conceptually:

hash(ORD-1001)
       ↓
   Partition 3

All events for that order are routed to the same logical partition.

Kafka can therefore preserve:

ORDER_CREATED
      ↓
ORDER_PAID
      ↓
ORDER_SHIPPED

for that business entity.

---

Do We Need Global Ordering?

Usually, no.

Events belonging to:

ORD-1001

may need ordering relative to one another.

But events for:

ORD-1001

and:

ORD-9008

usually do not need a global ordering relationship.

The scalable rule is:

Preserve ordering per business entity rather than trying to guarantee global ordering.

---

What Happens If a Kafka Broker Fails?

Kafka partitions are normally replicated across brokers.

Suppose:

Partition 3 Leader
→ Broker A

and Broker A fails.

Kafka can elect a replica:

Partition 3 New Leader
→ Broker B

The producer continues targeting the same logical partition.

The message key does not need to know which physical broker currently hosts the partition leader.

Conceptually:

hash(orderId)
      ↓
Partition
      ↓
Current Partition Leader

The physical broker can change while the logical partition remains the same.

---

Production Failure Scenario: CDC Connector Downtime

Suppose Debezium or AWS DMS stops at:

10:00 AM

and comes back at:

2:00 PM

The connector must still process the changes generated during those four hours.

This is where transaction-log retention becomes critical.

If the required WAL, Binlog, or Redo data is still available:

Connector Restarts
       ↓
Resume from Previous Position
       ↓
Read Missing Changes
       ↓
Catch Up

But if the required transaction logs have already been removed:

Required Changes Unavailable
       ↓
CDC Cannot Fully Recover
       ↓
Re-sync May Be Required

---

CDC Offset vs Transaction-Log Retention

These concepts are related but different.

The connector offset answers:

Where did I stop?

The transaction log answers:

What changes happened after that position?

Therefore:

Offset tells CDC where to resume; transaction-log retention ensures the required changes are still available.

---

Production Retention Strategy

Transaction-log retention should be based on the maximum CDC outage the system is expected to tolerate.

For example:

Maximum Expected CDC Outage
= 24 hours

Then the transaction-log retention needs to be:

> 24 hours + safety buffer

Longer retention improves the recovery window, but it also consumes additional database storage.

Do not choose the retention period arbitrarily.

---

Backpressure in CDC Pipelines

Backpressure occurs when events enter the pipeline faster than downstream consumers can process them.

Example:

Database Changes
10,000 events/sec

Consumer Capacity
5,000 events/sec

The remaining events accumulate:

Producer > Consumer
        ↓
Kafka Backlog Grows
        ↓
Consumer Lag Increases

Possible mitigation includes:

• scaling consumer instances
• increasing Kafka partitions
• batch processing
• optimizing downstream systems
• applying rate limiting where appropriate

---

Duplicate Events and Idempotency

CDC pipelines commonly operate using at-least-once delivery semantics.

That means an event can occasionally be delivered more than once.

Example:

ORDER_PAID
ORDER_PAID

Therefore downstream consumers should normally be idempotent.

One common technique is to store the processed eventId.

Example:

Processed_Event

event_id
--------
EVT-9001

If EVT-9001 arrives again:

Event already processed
→ Skip duplicate

This prevents the business operation from being applied twice.

---

Schema Evolution

CDC also introduces schema-evolution concerns.

Suppose the source orders table later gains:

discount_amount

Downstream consumers should not suddenly fail just because a new field was introduced.

Common practices include:

• backward-compatible event schemas
• optional fields
• versioned events
• Schema Registry
• consumer compatibility testing

Kafka architectures often use formats such as:

Avro
Protobuf
JSON Schema

with a Schema Registry to control compatibility.

---

Common CDC Production Problems

Problem	Impact	Typical Mitigation
Connector outage	CDC lag	Restart + sufficient log retention
WAL/Binlog removed	Missing recovery history	Larger retention window
Log growth	DB storage pressure	Monitor lag and disk
Traffic spike	Backpressure	Scale consumers / partitions
Duplicate events	Duplicate processing	Idempotent consumer
Schema change	Consumer failure	Schema evolution/versioning
Broker failure	Temporary disruption	Kafka replication
Slow consumers	Increasing lag	Scale / optimize
Bad events	Processing failure	Retry / DLQ
Ordering issue	Invalid business state	Business key as partition key

---

CDC vs Transactional Outbox

CDC and Outbox are not competing patterns.

They solve different parts of the problem.

CDC answers:

How do I reliably capture database changes?

Outbox answers:

What meaningful business event should I publish?

Together:

Business Transaction
        ↓
    Outbox Event
        ↓
 Transaction Log
        ↓
       CDC
        ↓
      Kafka

That distinction is one of the most important takeaways from this architecture.

---

Where Does AWS DMS Fit with Outbox?

AWS DMS can also capture changes made to an Outbox table.

For example:

Aurora PostgreSQL
       ↓
      WAL
       ↓
    AWS DMS
       ↓
  Amazon MSK

However, in long-running Kafka-centric streaming architectures, teams often choose Debezium because it is specifically designed around Kafka and CDC ecosystems.

For migration-oriented scenarios, AWS DMS is generally the more natural AWS-managed option.

---

Should We Delete Outbox Records?

An Outbox table can grow indefinitely if old records are never cleaned up.

A production strategy could look like:

CDC Publishes Event
       ↓
Retain Event Temporarily
       ↓
Archive or Delete Old Records

Retention can depend on:

• age
• processing status
• compliance requirements
• recovery requirements

The important rule is:

Do not remove Outbox events before you are confident the CDC pipeline has captured them.

---

What Should We Monitor in Production?

At minimum, monitor:

• CDC connector health
• CDC replication lag
• WAL / Binlog / Redo retention
• database disk usage
• Kafka producer failures
• Kafka consumer lag
• DLQ / failed events
• schema changes

A useful operational flow is:

Database
   ↓
Transaction Log
   ↓
CDC Connector
   ↓
Kafka
   ↓
Consumer

Monitor the health and lag at every boundary.

---

Final Production Architecture

---

End-to-End Mental Model

If you remember only one flow from this entire article, remember this:

1. Application executes business transaction
              ↓
2. Business row + Outbox event commit together
              ↓
3. Database writes the transaction to WAL/Binlog/Redo
              ↓
4. CDC connector reads the transaction log
              ↓
5. Outbox event is published to Kafka / Amazon MSK
              ↓
6. Related events use the business ID as Kafka key
              ↓
7. Consumers process events idempotently
              ↓
8. Lag, retention and failures are monitored

That is the complete production mental model.

---

Frequently Asked Interview Questions

1. What is Change Data Capture?

CDC captures INSERT, UPDATE and DELETE changes occurring in a database and propagates them to downstream systems, commonly by reading transaction logs.

2. What are WAL, Binlog and Redo Logs?

PostgreSQL uses WAL, MySQL uses Binlog, and Oracle uses Redo Logs / archived redo logs.

3. Why not continuously poll application tables?

Polling creates additional database load and makes reliable tracking of changes, deletes, and ordering more difficult.

4. AWS DMS or Debezium?

A simple rule:

Database Migration
→ AWS DMS

Kafka-Centric Continuous CDC
→ Debezium

5. What is Full Load + CDC?

First copy the existing historical records, then continuously replicate new changes.

6. What is the Transactional Outbox Pattern?

The application updates its business data and writes the corresponding business event to an Outbox table inside the same database transaction.

7. Does Outbox replace CDC?

No.

Outbox creates the meaningful business event.

CDC captures that event from the transaction log and moves it downstream.

8. How do we maintain Kafka ordering?

Use a stable business identifier such as:

orderId
customerId
accountId

as the Kafka message key so related events go to the same partition.

9. What happens if CDC goes down?

When it restarts, it resumes from its previous position as long as the required transaction logs are still retained.

10. How do we handle duplicate CDC events?

Use idempotent consumers and a stable identifier such as eventId.

11. What is backpressure?

Backpressure occurs when changes are produced faster than downstream consumers can process them.

12. What should we monitor?

Monitor CDC lag, connector health, log retention, database storage, Kafka lag, failed events and DLQ.

---

Key Takeaways

1. CDC captures database changes without continuously polling application tables.
2. PostgreSQL uses WAL, MySQL uses Binlog, and Oracle uses Redo Logs.
3. AWS DMS is particularly useful for migrations and Full Load + CDC.
4. Debezium is commonly used for continuous Kafka-centric CDC.
5. Plain CDC may expose low-level database changes rather than meaningful domain events.
6. The Transactional Outbox Pattern writes business state and the corresponding event in the same transaction.
7. Outbox does not replace CDC; the two patterns complement each other.
8. CDC captures the Outbox insert from the transaction log and publishes it downstream.
9. Kafka ordering should normally be maintained per business entity using an appropriate message key.
10. Consumers should be idempotent because duplicate delivery can occur.
11. Transaction-log retention must exceed the CDC recovery window.
12. Backpressure, connector lag, consumer lag and storage growth must be monitored.
13. Schema evolution should be managed through compatible schemas and versioning.
14. Outbox + CDC + Kafka forms a powerful pattern for building reliable event-driven architectures.