Idempotency

Status: Complete
Category: Architecture
Default enforcement: Soft
Author: PushBackLog team

Tags

• Topic: architecture, distributed-systems, reliability
• Skillset: backend
• Technology: generic
• Stage: execution, review

Summary

An idempotent operation produces the same result whether it is executed once or many times. Designing APIs and service operations to be idempotent is essential for safe retry behaviour in distributed systems: networks fail, clients timeout, and messages are delivered more than once. Without idempotency, retries cause duplicate charges, duplicate records, and double-processing. With it, retries are safe and systems are self-healing.

Rationale

Retries are mandatory in distributed systems

Any operation that crosses a network boundary can fail for reasons outside the caller’s control: transient network errors, upstream service restarts, client timeouts, DNS failures. The caller cannot know whether the failure occurred before, during, or after the recipient processed the request. The safe response is to retry. But retry is only safe if the operation is idempotent — otherwise, the retry may execute the operation twice with observable side effects.

Payment processing is the classic example. A client sends a charge request and receives a network timeout. Did the charge succeed before the timeout? If the API is not idempotent, retrying creates a duplicate charge. If it is, the retry returns the same result as the original request without re-processing.

At-least-once delivery is the default in modern queueing systems

SQS, Kafka, and most other message systems guarantee at-least-once delivery — a message will be delivered at least once, but may be delivered more than once (e.g., if a consumer crashes before acknowledging). Consumers must be idempotent by design. An event handler that is not idempotent will produce incorrect results whenever a message is redelivered.

Guidance

Idempotency keys for mutations

For APIs that perform state-changing operations (POST, PUT, DELETE), accept an optional client-generated idempotency key. Use the key to detect and deduplicate repeated requests.

POST /payments
Idempotency-Key: 7e1c9a4b-2d83-4f0a-99e2-f3d5a1e2b9c7

{
  "amount": 4999,
  "currency": "GBP",
  "customerId": "cus_abc123"
}

Server behaviour:

1. Receive request with Idempotency-Key
2. Look up the key in an idempotency store (Redis / DB)
3. If FOUND:
     - Return the stored response (do not re-process)
4. If NOT FOUND:
     - Process the request
     - Store the result against the key (with a TTL of 24–72 hours)
     - Return the result

async function createPayment(
  dto: CreatePaymentDto,
  idempotencyKey: string
): Promise<Payment> {
  const cached = await idempotencyStore.get(idempotencyKey);
  if (cached) {
    return cached as Payment; // Return stored result, do not re-charge
  }

  const payment = await paymentGateway.charge(dto);
  await idempotencyStore.set(idempotencyKey, payment, { ttl: 86400 });
  return payment;
}

Natural idempotency via PUT and conditional writes

HTTP PUT is semantically idempotent — setting a resource to a known state always produces the same result.

PUT /users/u_123/preferences
{ "theme": "dark", "notifications": false }

No matter how many times this request is sent, the user preferences end up in the same state.

For database upserts, use INSERT ... ON CONFLICT DO NOTHING or ORM equivalents:

// Upsert — idempotent regardless of how many times it is called
await db.user.upsert({
  where: { id: userId },
  create: userDto,
  update: userDto,
});

Idempotent message consumers

Deduplication at the consumer level:

async function handleOrderPlaced(event: OrderPlacedEvent): Promise<void> {
  const alreadyProcessed = await processedEvents.has(event.eventId);
  if (alreadyProcessed) {
    logger.info({ eventId: event.eventId }, 'Skipping duplicate event');
    return;
  }

  await fulfilmentService.startFulfilment(event.orderId);
  await processedEvents.add(event.eventId, { ttl: 86400 });
}

Use the message’s messageId or a business-level identifier (e.g., orderId + eventType) as the deduplication key.

Which operations must be idempotent

Design checklist

• All mutation endpoints accept an optional Idempotency-Key header
• Idempotency store (Redis or DB) is used to deduplicate requests within a TTL window
• Queue/event consumers check for already-processed event IDs before executing side effects
• DELETE endpoints return success (not 404) when the resource has already been deleted
• Database unique constraints enforce business-level idempotency where appropriate