Building Resilient Event-Driven Microservices: Idempotency, Outbox, and Retries

#The Dual-Write Problem in Microservices

Consider this ubiquitous anti-pattern:

// ❌ DANGEROUS ANTI-PATTERN: Dual-Write Vulnerability
async function processOrder(orderData: Order) {
  // 1. Write to database
  await db.orders.create({ data: orderData });

  // 2. Publish to message broker (Kafka / RabbitMQ)
  await kafkaProducer.send({ topic: 'orders.created', message: orderData });
}

What happens if the database write succeeds, but the network connection to Kafka drops or the process crashes right after? Your database and event stream are permanently out of sync.

Conversely, if you publish to Kafka first, what happens if the database transaction aborts due to a uniqueness constraint? You just emitted a ghost event that downstream services will act upon!

#The Solution: The Transactional Outbox Pattern

To eliminate dual-write hazards, database mutations and event publications must share the same ACID database transaction.

BEGIN;

-- 1. Mutate application domain state
INSERT INTO orders (id, customer_id, amount, status) 
VALUES ('ord_123', 'cust_456', 99.00, 'CONFIRMED');

-- 2. Stage event in outbox table within same transaction!
INSERT INTO outbox_events (id, aggregate_type, aggregate_id, payload, created_at)
VALUES (
  gen_random_uuid(), 
  'Order', 
  'ord_123', 
  '{"id":"ord_123","amount":99.00}', 
  NOW()
);

COMMIT;

A dedicated background relay process (such as Debezium reading Postgres Write-Ahead Logs or a polling worker with SELECT ... FOR UPDATE SKIP LOCKED) safely relays outbox events to the message broker.

#Idempotent Consumer Execution

Network networks are inherently unreliable: messages can be delivered multiple times. Every consumer must enforce idempotency.

async function handleEvent(event: DomainEvent) {
  const processed = await redis.set(
    `idempotency:event:${event.id}`,
    'PROCESSED',
    'EX',
    86400, // 24 hour TTL
    'NX'   // Only set if Not Exists
  );

  if (!processed) {
    console.log(`Duplicate event ${event.id} detected. Skipping.`);
    return;
  }

  await executeBusinessLogic(event);
}

By pairing the Transactional Outbox Pattern with Idempotency enforcement, distributed systems achieve rock-solid reliability under network partitions and service restarts.