Designing Idempotent APIs and Distributed Locks with Redis

A payments engineer I worked with once described their worst on-call night as “the morning we refunded people for orders we’d already refunded.” The root cause wasn’t fraud, and it wasn’t a bad deploy. It was a mobile client on flaky 4G that retried a POST /refund request three times because the first two responses timed out on the way back. The server had processed all three. Nothing in the stack had asked the obvious question: have I already done this?

That question is the entire premise of this post. The same failure shows up outside payments too, a claim submitted twice to a payer, or an appointment request that creates two bookings when a patient portal retries on a bad connection. Networks fail, clients retry, and multiple instances of your API are usually running at once behind a load balancer. Once you accept those three facts, two problems show up almost immediately: how do you make an operation safe to repeat, and how do you make sure two processes don’t step on the same piece of work at the same moment? Idempotency keys solve the first. Distributed locks solve the second. Redis, it turns out, is a genuinely good fit for both, mostly because of one unglamorous feature: atomic operations with expiry.

We’ll build a real idempotency layer and a real distributed lock, both backed by Redis and wired into an Express (and then NestJS) service. Along the way we’ll hit the mistakes that tend to bite teams the first time they try this in production, locks that never release, idempotency keys that cache the wrong thing, and the classic “I deleted someone else’s lock” bug.

Why Redis for this? Both idempotency and locking need one thing above all else: a single, fast, shared source of truth that every API instance can agree on right now. Redis’s single-threaded command execution, SET with NX/PX in one atomic call, and Lua scripting give us exactly that, without standing up a separate coordination service like ZooKeeper or etcd.

Prerequisites

• Node.js 20 LTS or later
• Redis 7.x running locally (docker run -p 6379:6379 redis:7-alpine)
• ioredis, npm install ioredis
• Comfort with TypeScript generics and async/await
• A basic sense of HTTP status codes, particularly 409 Conflict and 202 Accepted

The Problem Space

What “Idempotent” Actually Means Here

Textbook idempotency says calling an operation once or five times produces the same result. PUT and DELETE are supposed to be idempotent by the HTTP spec; POST is not. In practice, though, most of the POST endpoints that cause real damage when duplicated, charge a card, send an email, create an order, need to behave as if they were idempotent even though the spec doesn’t require it. The usual fix is an Idempotency-Key header the client generates once per logical operation and resends on every retry. The server’s job is to recognize a repeat key and return the original result instead of doing the work twice.

The Retry Trap

Here’s the part that catches teams off guard: the danger window isn’t just “the client sent the same request twice on purpose.” It’s the request that timed out on the way back to the client after the server had already finished it. The client has no way to know whether the write succeeded, so a well-behaved client retries. If your server can’t tell “new request” from “same request, second delivery,” it will happily do the work again.

Enter Distributed Locks

Idempotency keys protect a single logical operation from being repeated. Distributed locks protect a shared resource from being touched by two different operations at the same time, think two workers both trying to decrement the same inventory count, or a cron job that must never run on two nodes simultaneously. They solve adjacent but different problems, and mixing them up is a common source of half-working solutions.

Idempotency Keys with Redis

The Intuition

The approach is simpler than most people expect. When a request arrives carrying an Idempotency-Key header, we try to atomically claim that key in Redis. If we’re the first to claim it, we proceed, run the handler, and store the response against the key. If someone already claimed it, including “someone” being an earlier attempt from the very same client, we skip the handler entirely and return the stored response, or a 409 if the original request is still in flight.

Redis Data Model

Each idempotency key maps to a single Redis hash with three fields: status (pending or completed), a hash of the original request body (so a reused key with a different payload is rejected rather than silently accepted), and the serialized response once it’s available. We wrap the claim in a Lua script so the check-and-set can’t race across two API instances handling the retry at nearly the same instant.

The Lua Script

-- idempotency_claim.lua
local key = KEYS[1]
local requestHash = ARGV[1]
local ttlMs = tonumber(ARGV[2])
local existing = redis.call('HGET', key, 'requestHash')
if existing == false then
  -- first time we've ever seen this idempotency key
  redis.call('HSET', key, 'status', 'pending', 'requestHash', requestHash)
  redis.call('PEXPIRE', key, ttlMs)
  return {'CLAIMED'}
end
if existing ~= requestHash then
  -- same key, different payload -- reject, this is a client bug
  return {'MISMATCH'}
end
local status = redis.call('HGET', key, 'status')
if status == 'pending' then
  return {'IN_FLIGHT'}
end
local response = redis.call('HGET', key, 'response')
return {'REPLAY', response}

Node.js Implementation

// idempotency-store.ts
import Redis from 'ioredis';
import crypto from 'crypto';
import fs from 'fs';
import path from 'path';
type ClaimResult =
  | { kind: 'claimed' }
  | { kind: 'mismatch' }
  | { kind: 'in_flight' }
  | { kind: 'replay'; response: unknown };
export class IdempotencyStore {
  private sha: string | null = null;
  constructor(
    private redis: Redis,
    private ttlMs: number = 24 * 60 * 60 * 1000,
  ) {}
  private async load(): Promise {
    const script = fs.readFileSync(
      path.join(__dirname, 'idempotency_claim.lua'), 'utf8',
    );
    this.sha = await this.redis.script('LOAD', script);
  }
  hashBody(body: unknown): string {
    return crypto.createHash('sha256').update(JSON.stringify(body)).digest('hex');
  }
  async claim(idempotencyKey: string, body: unknown): Promise {
    if (!this.sha) await this.load();
    const requestHash = this.hashBody(body);
    const key = `idem:${idempotencyKey}`;
    const result = await this.redis.evalsha(
      this.sha!, 1, key, requestHash, String(this.ttlMs),
    ) as string[];
    switch (result[0]) {
      case 'CLAIMED': return { kind: 'claimed' };
      case 'MISMATCH': return { kind: 'mismatch' };
      case 'IN_FLIGHT': return { kind: 'in_flight' };
      case 'REPLAY':
        return { kind: 'replay', response: JSON.parse(result[1]) };
      default:
        throw new Error(`Unexpected claim result: ${result[0]}`);
    }
  }
  async complete(idempotencyKey: string, response: unknown): Promise {
    const key = `idem:${idempotencyKey}`;
    await this.redis.hset(key, 'status', 'completed', 'response', JSON.stringify(response));
  }
}

Express Middleware

import { Request, Response, NextFunction } from 'express';
export function idempotencyMiddleware(store: IdempotencyStore) {
  return async (req: Request, res: Response, next: NextFunction) => {
    const key = req.header('Idempotency-Key');
    if (!key) return next(); // header is opt-in, not mandatory here
    const result = await store.claim(key, req.body);
    if (result.kind === 'mismatch') {
      return res.status(422).json({ error: 'Idempotency-Key reused with a different payload' });
    }
    if (result.kind === 'in_flight') {
      return res.status(409).json({ error: 'Request with this key is still processing' });
    }
    if (result.kind === 'replay') {
      return res.status(200).json(result.response);
    }
    // 'claimed' -- run the real handler, then persist the response
    const originalJson = res.json.bind(res);
    res.json = (body: unknown) => {
      store.complete(key, body).catch(() => {});
      return originalJson(body);
    };
    next();
  };
}

One detail worth calling out: we deliberately reject a reused key attached to a different payload instead of quietly serving the old response. That’s almost always a client-side bug, someone hashed the wrong field, or reused a UUID across two different button clicks, and silently papering over it just moves the confusion further downstream.

Distributed Locks with Redis

The Intuition

A distributed lock is mutual exclusion stretched across processes instead of threads. Only one holder at a time, everyone else waits or backs off, and, critically, the lock has to release itself even if the holder crashes mid-operation. That last requirement is exactly what a plain in-memory mutex can’t give you and Redis can, via key expiry.

Why a Bare SETNX Isn’t Enough

The naive version looks tempting: SET lock:orders:42 1 NX EX 10, and if it succeeds, you hold the lock. It mostly works, right up until two things go wrong. First, if the process that holds the lock is slow, a GC pause, a downstream call that hangs, the lock can expire while the work is still running, and a second process grabs it, so now two workers are mutating the same order concurrently, which is the exact scenario the lock existed to prevent. Second, when it’s time to release the lock, a naive DEL will happily delete a lock that isn’t yours anymore, because it already expired and someone else claimed it.

The Lock Token Pattern

The fix for the second problem is to never blindly DEL. Each lock acquisition gets a random token, stored as the value. Releasing the lock means comparing the stored value against your token and only deleting if they match, atomically, via Lua, because a plain GET-then-DEL from application code is itself a race condition.

The Lua Script

-- release_lock.lua
-- KEYS[1] = lock key, ARGV[1] = token this client holds
if redis.call('GET', KEYS[1]) == ARGV[1] then
  return redis.call('DEL', KEYS[1])
else
  return 0 -- someone else's lock now, or it already expired
end

Node.js Implementation

// redis-lock.ts
import Redis from 'ioredis';
import crypto from 'crypto';
import fs from 'fs';
import path from 'path';
export class RedisLock {
  private releaseSha: string | null = null;
  constructor(private redis: Redis) {}
  private async loadRelease(): Promise {
    const script = fs.readFileSync(
      path.join(__dirname, 'release_lock.lua'), 'utf8',
    );
    this.releaseSha = await this.redis.script('LOAD', script);
  }
  async acquire(
    resource: string,
    ttlMs: number,
    retries = 3,
    retryDelayMs = 100,
  ): Promise<{ token: string } | null> {
    const key = `lock:${resource}`;
    const token = crypto.randomBytes(16).toString('hex');
    for (let attempt = 0; attempt <= retries; attempt++) { const ok = await this.redis.set(key, token, 'PX', ttlMs, 'NX'); if (ok === 'OK') return { token }; await new Promise(r => setTimeout(r, retryDelayMs * (attempt + 1)));
    }
    return null; // gave up -- caller decides whether that's fatal
  }
  async release(resource: string, token: string): Promise {
    if (!this.releaseSha) await this.loadRelease();
    const key = `lock:${resource}`;
    const result = await this.redis.evalsha(this.releaseSha!, 1, key, token);
    return result === 1;
  }
}

Auto-Extending Locks for Long-Running Work

A fixed TTL forces an awkward guess: too short and legitimate slow work loses its lock mid-flight; too long and a crashed holder blocks everyone else for that whole window. The common fix is a watchdog, a timer that extends the TTL periodically for as long as the holder is genuinely still alive and working, and stops the moment the work finishes or the process dies.

async function withLock(
  lock: RedisLock,
  resource: string,
  ttlMs: number,
  fn: () => Promise,
): Promise {
  const held = await lock.acquire(resource, ttlMs);
  if (!held) throw new Error(`Could not acquire lock for ${resource}`);
  const watchdog = setInterval(() => {
    lock.redis.pexpire(`lock:${resource}`, ttlMs).catch(() => {});
  }, Math.floor(ttlMs / 2));
  try {
    return await fn();
  } finally {
    clearInterval(watchdog);
    await lock.release(resource, held.token);
  }
}

On Redlock

If you’re locking across a multi-node Redis deployment rather than a single primary, the single-instance pattern above isn’t sufficient, look at the Redlock algorithm, which acquires the lock against a majority of independent Redis nodes. It’s a genuinely different set of trade-offs and, notably, a source of long-running disagreement even among distributed-systems people about how strong its guarantees actually are. Worth reading both sides before you adopt it for anything safety-critical.

Side-by-Side Comparison

Idempotency keys and distributed locks solve neighboring problems and sometimes get reached for interchangeably. They shouldn’t be.

Common Pitfalls

Caching the Wrong Thing

Only mark an idempotency key as completed once you have a response you’re willing to replay forever (or until the TTL). If the handler throws partway through, delete the pending key rather than leaving it stuck, otherwise every retry gets stonewalled with a 409 for no good reason.

Forgetting the TTL on a Lock

A lock with no expiry is a deadlock waiting for a deploy. Always set one, and set it generously enough that normal execution finishes comfortably inside it, the watchdog pattern above exists specifically so you don’t have to pick between “too short” and “too long.”

Skipping the Token Check on Release

This is the bug that’s hardest to catch in testing because it only shows up under real load: process A’s lock expires under heavy GC pressure, process B acquires it, and then process A finishes its work and calls DEL, wiping out process B’s lock. The token comparison in the Lua script above exists purely to stop this.

Treating Idempotency Keys as a Substitute for Database Constraints

Redis-backed idempotency is a fast, request-level guard, it is not a replacement for a unique constraint on your orders or payments table. Keep both. In a claims or orders schema this might be a unique constraint on (payer_id, claim_reference), the same defense-in-depth logic applies whether the record is a retail order or a claim line. The Redis layer stops the duplicate request from reaching your business logic in the common case; the database constraint is the backstop for the case it misses, like a Redis eviction under memory pressure.

Caching PHI Inside the Idempotency Store

The cached response lives in Redis for the full TTL, which can be hours or days. If that response contains patient data, that’s PHI at rest for the whole window. Worth being deliberate about what gets stored, a reference/ID versus the full payload, and how that Redis instance is secured, rather than caching the raw response by default.

When to Choose Which

Reach for Idempotency Keys When:

• The endpoint has side effects a client might trigger more than once, payments, order creation, notification sends
• Clients (mobile apps especially) are on unreliable networks and will retry on timeout
• You need to safely support at-least-once delivery from a queue or webhook sender
• The operation’s result is worth caching and replaying verbatim on retry
• In healthcare systems, claims submissions, prior-authorization requests, prescription refill requests. A retried request here doesn’t just annoy a customer, it can double-bill a payer or double-queue a clinical action.

Reach for Distributed Locks When:

• Multiple workers or API instances might touch the same resource concurrently
• A scheduled job must run on exactly one node, not every node in the cluster
• You’re protecting a critical section that a database transaction alone can’t cover, for example, a call that spans two different services
• The operation has a bounded, predictable duration you can size a TTL against
• A nightly claims batch or eligibility-reconciliation job needs to run on exactly one node, running it twice can generate duplicate claim submissions to a payer

Rule of thumb

If the danger is “the same request twice,” reach for idempotency keys. If the danger is “two different things at the same time,” reach for a lock. Plenty of production systems that started fine on the payment path (idempotency only) eventually needed a lock too, once a batch reconciliation job started running alongside live traffic.

NestJS Integration

Both patterns drop cleanly behind an Injectable provider, the same shape as any other cross-cutting NestJS concern.

The Idempotency Interceptor

// idempotency.interceptor.ts
import {
  Injectable, NestInterceptor, ExecutionContext, CallHandler, ConflictException,
} from '@nestjs/common';
import { Observable, of } from 'rxjs';
import { tap } from 'rxjs/operators';
import { IdempotencyStore } from './idempotency-store';
@Injectable()
export class IdempotencyInterceptor implements NestInterceptor {
  constructor(private readonly store: IdempotencyStore) {}
  async intercept(ctx: ExecutionContext, next: CallHandler): Promise<Observable> {
    const req = ctx.switchToHttp().getRequest();
    const key = req.headers['idempotency-key'];
    if (!key) return next.handle();
    const result = await this.store.claim(key, req.body);
    if (result.kind === 'in_flight') {
      throw new ConflictException('Request with this key is already processing');
    }
    if (result.kind === 'replay') return of(result.response);
    return next.handle().pipe(
      tap(response => this.store.complete(key, response)),
    );
  }
}

The Lock Provider

Applying @UseInterceptors(IdempotencyInterceptor) on a controller method is the whole integration, no changes to the handler’s business logic. The RedisLock class from earlier registers the same way, as a provider injected wherever a critical section needs it, keeping the coordination logic out of the service methods that use it.

Testing Both Patterns

The tricky part of testing locks and idempotency isn’t the happy path, it’s proving the failure modes actually fail safely. For idempotency, that means asserting a second request with the same key never re-executes the handler. For locks, it means asserting a second acquire attempt genuinely blocks until the first is released, not just that it succeeds eventually.

// idempotency behaves correctly on retry
const handlerSpy = jest.fn().mockResolvedValue({ orderId: 'abc' });
await request(app).post('/orders')
  .set('Idempotency-Key', 'client-key-1')
  .send({ sku: 'WIDGET-1' });
await request(app).post('/orders')
  .set('Idempotency-Key', 'client-key-1')
  .send({ sku: 'WIDGET-1' });
expect(handlerSpy).toHaveBeenCalledTimes(1); // not 2
// lock genuinely serializes two concurrent holders
const lock = new RedisLock(redis);
const first = await lock.acquire('inventory:sku-1', 5000);
const second = await lock.acquire('inventory:sku-1', 5000, 0);
expect(first).not.toBeNull();
expect(second).toBeNull(); // correctly blocked while first still holds it