Designing Idempotent APIs for Distributed Systems

The Problem

In a distributed system, things fail. Networks drop. Lambda functions time out. Clients retry. If your API processes a payment twice because of a retry, you have a very angry customer.

Idempotency means that making the same request multiple times produces the same result as making it once. It's not optional in distributed systems — it's a requirement.

Strategy 1: Idempotency Keys

The client sends a unique key with each request. The server uses it to detect duplicates.

Node.js Implementation

import { Pool } from "pg";

const pool = new Pool();

async function processPayment(
  idempotencyKey: string,
  payload: PaymentRequest
): Promise<PaymentResult> {
  const client = await pool.connect();
  try {
    await client.query("BEGIN");

    // Check if we've seen this key before
    const existing = await client.query(
      `SELECT result FROM idempotency_keys
       WHERE key = $1 AND created_at > NOW() - INTERVAL '24 hours'`,
      [idempotencyKey]
    );

    if (existing.rows.length > 0) {
      // Return the cached result
      return JSON.parse(existing.rows[0].result);
    }

    // Process the payment
    const result = await chargeCustomer(client, payload);

    // Store the result for future duplicate checks
    await client.query(
      `INSERT INTO idempotency_keys (key, result, created_at)
       VALUES ($1, $2, NOW())`,
      [idempotencyKey, JSON.stringify(result)]
    );

    await client.query("COMMIT");
    return result;
  } catch (e) {
    await client.query("ROLLBACK");
    throw e;
  } finally {
    client.release();
  }
}

The Database Table

CREATE TABLE idempotency_keys (
  key TEXT PRIMARY KEY,
  result JSONB NOT NULL,
  created_at TIMESTAMP NOT NULL DEFAULT NOW()
);

-- Auto-cleanup old keys
CREATE INDEX idx_idempotency_keys_created
ON idempotency_keys (created_at);

Run a periodic job to delete keys older than 24-48 hours.

Strategy 2: Natural Idempotency

Some operations are naturally idempotent:

// Naturally idempotent: SET is the same regardless of how many times you call it
await db.query(
  "UPDATE users SET email = $1 WHERE id = $2",
  [newEmail, userId]
);

// NOT idempotent: INCREMENT changes with every call
await db.query(
  "UPDATE accounts SET balance = balance + $1 WHERE id = $2",
  [amount, accountId]
);

Prefer SET over INCREMENT. If you must increment, use an idempotency key or a transaction log.

Strategy 3: Conditional Writes

Use optimistic locking to prevent duplicate processing:

async function fulfillOrder(orderId: string): Promise<boolean> {
  const result = await db.query(
    `UPDATE orders
     SET status = 'fulfilled', fulfilled_at = NOW()
     WHERE id = $1 AND status = 'pending'
     RETURNING id`,
    [orderId]
  );

  // If no rows were updated, the order was already fulfilled
  return result.rowCount > 0;
}

The WHERE status = 'pending' clause ensures only the first call succeeds. Subsequent calls are no-ops.

Strategy 4: SQS Deduplication

For FIFO queues, SQS handles deduplication natively:

import boto3
import hashlib

sqs = boto3.client("sqs")

def send_order_event(order_id: str, payload: dict):
    sqs.send_message(
        QueueUrl=FIFO_QUEUE_URL,
        MessageBody=json.dumps(payload),
        MessageGroupId=order_id,
        MessageDeduplicationId=hashlib.sha256(
            f"{order_id}:{payload['action']}".encode()
        ).hexdigest(),
    )

SQS deduplicates messages with the same MessageDeduplicationId within a 5-minute window.

Applying to Webhooks

Webhook providers (Stripe, GitHub, etc.) retry on failure. Your webhook handler must be idempotent:

app.post("/webhooks/stripe", async (req, res) => {
  const event = req.body;

  // Use the event ID as the idempotency key
  const processed = await db.query(
    "SELECT 1 FROM processed_webhooks WHERE event_id = $1",
    [event.id]
  );

  if (processed.rows.length > 0) {
    return res.status(200).json({ received: true }); // Already processed
  }

  await handleStripeEvent(event);

  await db.query(
    "INSERT INTO processed_webhooks (event_id) VALUES ($1) ON CONFLICT DO NOTHING",
    [event.id]
  );

  res.status(200).json({ received: true });
});

Testing Idempotency

describe("processPayment", () => {
  it("returns the same result for duplicate requests", async () => {
    const key = "test-key-123";
    const payload = { amount: 100, currency: "USD", customerId: "cust_1" };

    const result1 = await processPayment(key, payload);
    const result2 = await processPayment(key, payload);

    expect(result1).toEqual(result2);
    // Verify charge was made exactly once
    expect(mockChargeCustomer).toHaveBeenCalledTimes(1);
  });
});

Checklist

Before shipping an API endpoint, ask:

• What happens if this request is sent twice?
• What happens if the client retries after a timeout (the first request may have succeeded)?
• Are database writes conditional or absolute?
• Do downstream side effects (emails, webhooks, charges) have deduplication?

Conclusion

Idempotency is a design decision, not an afterthought. Pick the right strategy for each endpoint: idempotency keys for complex operations, conditional writes for state transitions, and natural idempotency where possible. Your future self debugging a production incident at 2 AM will thank you.