Idempotency in Distributed Systems: Why It Matters and How to Implement It

A year ago, a customer reported being charged twice for the same order. Our payment logs showed two successful charges, 3.2 seconds apart, for the exact same amount, the same card, the same order. What happened? The client's network briefly dropped after the payment request was sent but before the response arrived. The client retried automatically (our mobile app had retry-on-timeout logic), and the payment service processed it as a brand-new request. Two charges, one order, one very unhappy customer.

This is the classic distributed systems failure mode that idempotency solves. In any system where network calls can fail, timeout, or be retried, you need guarantees that repeating an operation produces the same result as performing it once. Without idempotency, every retry is a potential duplicate — and in payment processing, duplicate means real money lost.

According to Stripe's documentation, their API processes idempotent retries for approximately 1.5% of all API requests — millions of retries per day that would have caused duplicate charges without idempotent handling. This guide explains idempotency from first principles, covers four implementation patterns, and gives you production-ready code for the most common scenarios.

What Is Idempotency? (The Precise Definition)

An operation is idempotent if performing it multiple times has the same effect as performing it once. Mathematically: f(f(x)) = f(x).

The HTTP/1.1 specification (RFC 7231) defines GET, PUT, and DELETE as idempotent by specification. POST is explicitly not idempotent — making POST requests idempotent requires application-level implementation.

Why Idempotency Matters in Distributed Systems

In a distributed system, three things can go wrong with any network call:

1. The request never reaches the server. Safe to retry — the operation never happened.
2. The request reaches the server and succeeds, but the response is lost. Dangerous to retry — the operation already happened, but the client doesn't know.
3. The request reaches the server and fails. Usually safe to retry — depends on the failure mode.

Case #2 is the killer. The client sees a timeout or connection error and retries, but the server already processed the request. Without idempotency, you get duplicates. This isn't theoretical — a 2023 AWS Builders' Library article documents that network retries are the #1 cause of duplicate processing in cloud-native applications.

Real-World Consequences of Missing Idempotency

• Payment processing: Customer charged twice → refund required, trust damaged
• Order placement: Two orders created → inventory mismatch, shipping confusion
• Email sending: Same email sent twice → spam complaints, unsubscribes
• Inventory updates: "Decrement by 1" applied twice → negative stock
• Account creation: Two accounts with same email → data integrity violation

Implementation Pattern 1: Idempotency Keys

The most common pattern, used by Stripe, PayPal, and most payment APIs. The client generates a unique key (UUID) for each logical operation and sends it with the request. The server stores the key and its result; subsequent requests with the same key return the stored result without re-executing the operation.

// Server-side idempotency middleware
const idempotencyMiddleware = async (req, res, next) => {
  const idempotencyKey = req.headers['idempotency-key'];

  if (!idempotencyKey) {
    return next(); // No key — process normally (non-idempotent)
  }

  // Check if we've seen this key before
  const existing = await db.query(
    'SELECT response_status, response_body, created_at FROM idempotency_keys WHERE key = $1',
    [idempotencyKey]
  );

  if (existing.rows.length > 0) {
    const cached = existing.rows[0];

    // Verify the request hasn't expired (e.g., 24 hours)
    const ageHours = (Date.now() - new Date(cached.created_at)) / 3600000;
    if (ageHours > 24) {
      await db.query('DELETE FROM idempotency_keys WHERE key = $1', [idempotencyKey]);
      return next(); // Expired — process as new
    }

    // Return cached response
    logger.info({ idempotencyKey }, 'Returning cached idempotent response');
    return res.status(cached.response_status).json(JSON.parse(cached.response_body));
  }

  // Lock this key (prevent concurrent requests with same key)
  try {
    await db.query(
      "INSERT INTO idempotency_keys (key, status, created_at) VALUES ($1, 'processing', NOW())",
      [idempotencyKey]
    );
  } catch (err) {
    if (err.code === '23505') { // Unique violation — concurrent request
      return res.status(409).json({ error: 'Request is already being processed' });
    }
    throw err;
  }

  // Capture the response
  const originalJson = res.json.bind(res);
  res.json = async (body) => {
    // Store the response for future identical requests
    await db.query(
      'UPDATE idempotency_keys SET status = $2, response_status = $3, response_body = $4 WHERE key = $1',
      [idempotencyKey, 'completed', res.statusCode, JSON.stringify(body)]
    );
    return originalJson(body);
  };

  next();
};

// Usage
app.post('/api/payments', idempotencyMiddleware, async (req, res) => {
  // This handler runs at most once per idempotency key
  const payment = await paymentService.charge(req.body);
  res.status(201).json(payment);
});

Client-Side Implementation

// Client generates idempotency key per logical operation
import { v4 as uuidv4 } from 'uuid';

async function createPayment(orderData) {
  const idempotencyKey = uuidv4(); // One key per payment attempt

  // Retry with the SAME key on failure
  for (let attempt = 0; attempt < 3; attempt++) {
    try {
      const response = await fetch('/api/payments', {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          'Idempotency-Key': idempotencyKey // Same key on every retry!
        },
        body: JSON.stringify(orderData)
      });

      if (response.ok) return response.json();
      if (response.status === 409) {
        // Request in progress — wait and retry
        await sleep(1000);
        continue;
      }
      throw new Error(`HTTP ${response.status}`);
    } catch (err) {
      if (attempt === 2) throw err;
      await sleep(Math.pow(2, attempt) * 1000); // Exponential backoff
    }
  }
}

Implementation Pattern 2: Natural Idempotency Keys

Sometimes the operation itself contains a natural idempotency key — a unique identifier that makes duplicates detectable without an explicit key header.

// Natural idempotency: use order_id as the key
async function processPayment(orderId, amount, cardToken) {
  // Try to insert payment — unique constraint on order_id prevents duplicates
  try {
    const result = await db.query(`
      INSERT INTO payments (order_id, amount, card_token, status, created_at)
      VALUES ($1, $2, $3, 'pending', NOW())
      ON CONFLICT (order_id) DO NOTHING
      RETURNING *
    `, [orderId, amount, cardToken]);

    if (result.rows.length === 0) {
      // Payment already exists for this order — return existing
      const existing = await db.query('SELECT * FROM payments WHERE order_id = $1', [orderId]);
      return { payment: existing.rows[0], wasRetry: true };
    }

    // Process the new payment
    const chargeResult = await stripeClient.charges.create({ amount, source: cardToken });
    await db.query('UPDATE payments SET status = $1, stripe_charge_id = $2 WHERE order_id = $3',
      ['succeeded', chargeResult.id, orderId]);

    return { payment: { ...result.rows[0], status: 'succeeded' }, wasRetry: false };

  } catch (err) {
    await db.query('UPDATE payments SET status = $1, error = $2 WHERE order_id = $3',
      ['failed', err.message, orderId]);
    throw err;
  }
}

Implementation Pattern 3: Conditional Updates (Optimistic Concurrency)

For update operations, use version numbers or ETags to ensure the update is only applied once:

// Optimistic concurrency with version numbers
async function updateInventory(productId, newQuantity, expectedVersion) {
  const result = await db.query(`
    UPDATE products
    SET quantity = $1, version = version + 1
    WHERE id = $2 AND version = $3
    RETURNING *
  `, [newQuantity, productId, expectedVersion]);

  if (result.rows.length === 0) {
    throw new ConflictError('Product was modified by another request — retry with current version');
  }

  return result.rows[0];
}

// API endpoint
app.put('/api/products/:id', async (req, res) => {
  const { quantity } = req.body;
  const expectedVersion = parseInt(req.headers['if-match']); // ETag-style

  try {
    const product = await updateInventory(req.params.id, quantity, expectedVersion);
    res.setHeader('ETag', product.version);
    res.json(product);
  } catch (err) {
    if (err instanceof ConflictError) {
      res.status(409).json({ error: err.message });
    } else {
      throw err;
    }
  }
});

Implementation Pattern 4: Idempotent Consumers (Message Queues)

Message queues deliver messages at-least-once. Your consumers must handle duplicates:

// Idempotent message consumer
async function processMessage(message) {
  const messageId = message.properties.messageId;

  // Atomic deduplication check
  const result = await db.query(`
    INSERT INTO processed_messages (message_id, status, started_at)
    VALUES ($1, 'processing', NOW())
    ON CONFLICT (message_id) DO NOTHING
    RETURNING message_id
  `, [messageId]);

  if (result.rows.length === 0) {
    // Already processed — skip
    logger.info({ messageId }, 'Duplicate message, skipping');
    return;
  }

  try {
    // Process the message (business logic)
    const data = JSON.parse(message.content);
    await handleOrderCreated(data);

    // Mark as completed
    await db.query(
      "UPDATE processed_messages SET status = 'completed', completed_at = NOW() WHERE message_id = $1",
      [messageId]
    );
  } catch (err) {
    await db.query(
      "UPDATE processed_messages SET status = 'failed', error = $1 WHERE message_id = $2",
      [err.message, messageId]
    );
    throw err; // Let the queue retry
  }
}

The Idempotency Database Table

-- Idempotency keys table
CREATE TABLE idempotency_keys (
  key VARCHAR(255) PRIMARY KEY,
  status VARCHAR(20) NOT NULL DEFAULT 'processing',
  request_path VARCHAR(500),
  request_method VARCHAR(10),
  response_status INTEGER,
  response_body JSONB,
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  completed_at TIMESTAMPTZ,
  expires_at TIMESTAMPTZ DEFAULT NOW() + INTERVAL '24 hours'
);

-- Index for cleanup
CREATE INDEX idx_idempotency_expires ON idempotency_keys(expires_at);

-- Cleanup job (run daily)
DELETE FROM idempotency_keys WHERE expires_at < NOW();

-- Processed messages table (for queue consumers)
CREATE TABLE processed_messages (
  message_id VARCHAR(255) PRIMARY KEY,
  status VARCHAR(20) NOT NULL DEFAULT 'processing',
  started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  completed_at TIMESTAMPTZ,
  error TEXT
);

-- Cleanup old messages (run daily)
DELETE FROM processed_messages WHERE started_at < NOW() - INTERVAL '7 days';

My Opinionated Idempotency Rules

1. Every POST endpoint that creates resources or triggers side effects must support idempotency keys. This is non-negotiable for payment, order, and user creation APIs. If you build a payment API without idempotency keys, you will have double-charge incidents — it's a matter of when, not if.

2. The client must generate the idempotency key, not the server. The whole point is that the client knows "this is the same logical operation" even after a timeout. If the server generates the key, the client can't retry with the same key because it never received the key.

3. Use UUIDs, not timestamps or sequential IDs. Timestamps are not unique (two requests in the same millisecond). Sequential IDs require coordination. UUIDs are unique without coordination — exactly what you need.

4. Set an expiration on idempotency keys. Without cleanup, your idempotency table grows forever. 24-48 hours is a reasonable TTL for most use cases. If a client retries after 48 hours, it's a new logical operation.

5. Make your database operations naturally idempotent where possible. INSERT ... ON CONFLICT DO NOTHING, UPDATE ... SET status = 'completed' WHERE status = 'pending', and conditional updates with version numbers are inherently idempotent without needing a separate key table.

Action Plan

Week 1: Audit and Identify

• List every API endpoint that modifies state (POST, PUT, PATCH, DELETE)
• For each, answer: "What happens if this is called twice with the same data?"
• Prioritize: payment endpoints first, then order creation, then everything else

Week 2: Implement

• Create the idempotency_keys table
• Build the idempotency middleware (or use a library)
• Add Idempotency-Key header support to priority endpoints
• Update client code to generate and send idempotency keys on retries

Week 3: Test and Monitor

• Write integration tests that simulate network failures and retries
• Add monitoring for idempotent cache hits (duplicate detection rate)
• Set up cleanup jobs for expired idempotency keys
• Document the idempotency pattern for the team

Sources and Further Reading

• Stripe — Idempotent Requests Documentation
• AWS Builders' Library — Making Retries Safe with Idempotent APIs
• RFC 7231 — HTTP/1.1 Semantics and Content
• Brandur Leach — Implementing Stripe-like Idempotency Keys
• microservices.io — Idempotent Consumer Pattern
• Martin Kleppmann — Designing Data-Intensive Applications

I'm Ismat, and I build BirJob — Azerbaijan's job aggregator scraping 80+ sources daily.