Exactly-Once Processing Is a Myth
(Here's What Actually Works)

The Two Generals Problem:

This is a classic thought experiment in distributed systems. Two armies need to coordinate an attack, but their only communication is through messengers who might be captured. Even if General A receives confirmation from General B, General B doesn't know if General A received that confirmation. This requires another confirmation, which itself needs confirmation—an infinite loop. There is no algorithm that guarantees both parties know the message was received in an unreliable network.

This fundamental limitation of distributed systems is why exactly-once delivery is impossible at the network level.

What Kafka Actually Guarantees:

• Idempotent Producers: Kafka assigns each message a sequence number. If the producer retries, Kafka deduplicates based on this number.
• Transactional Reads/Writes: Kafka can write multiple messages atomically across partitions and read them transactionally.
• Consumer Group Offset Management: Offsets are committed transactionally with processing, preventing double-reads.

The boundary: This only works inside Kafka. Once your consumer calls an external API, sends an email, or updates a database, you need application-level idempotency.

Pattern 1: Redis-Based Idempotency (Node.js)

const redis = require('redis');
const client = redis.createClient();

async function executeIdempotent(key, action) {
  // Check if we've seen this key before
  const cached = await client.get(key);
  if (cached) {
    return JSON.parse(cached); // Return cached result
  }

  // Execute the action
  const result = await action();

  // Store result with 24h TTL
  await client.setex(key, 86400, JSON.stringify(result));

  return result;
}

// Usage in payment handler
app.post('/checkout', async (req, res) => {
  const idempotencyKey = req.headers['idempotency-key'];

  const result = await executeIdempotent(idempotencyKey, async () => {
    // This only runs once, even if retried
    const charge = await stripe.charges.create({
      amount: req.body.amount,
      currency: 'usd',
      source: req.body.token
    });

    await db.orders.create({
      chargeId: charge.id,
      userId: req.user.id
    });

    return { orderId: charge.id, status: 'success' };
  });

  res.json(result);
});

Pattern 2: Database-Based Idempotency (Python)

from datetime import datetime, timedelta
import json

def execute_idempotent(db, key, action):
    # Try to fetch existing result
    result = db.query(
        "SELECT result FROM idempotency_keys WHERE key = %s",
        (key,)
    ).first()

    if result:
        return json.loads(result['result'])

    # Execute action
    action_result = action()

    # Store with timestamp
    db.execute(
        """INSERT INTO idempotency_keys (key, result, created_at)
           VALUES (%s, %s, %s)
           ON CONFLICT (key) DO NOTHING""",
        (key, json.dumps(action_result), datetime.now())
    )

    # Clean old keys (optional background job)
    db.execute(
        "DELETE FROM idempotency_keys WHERE created_at < %s",
        (datetime.now() - timedelta(days=1),)
    )

    return action_result

# Usage in webhook handler
@app.route('/webhooks/stripe', methods=['POST'])
def stripe_webhook():
    event = request.json
    event_id = event['id']  # Stripe's unique event ID

    result = execute_idempotent(db, event_id, lambda: {
        'user_id': activate_subscription(event['data']['object']),
        'status': 'activated'
    })

    return jsonify(result)

Pattern 3: Using OnceOnly API

// Delegate duplicate detection to OnceOnly (check-lock)
const key = `order-${userId}-${sessionId}`;

const lockRes = await fetch("https://api.onceonly.tech/v1/check-lock", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.ONCEONLY_API_KEY}`, // once_live_***
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    key,
    ttl: 3600,
    metadata: { userId, sessionId }
  }),
});

const lock = await lockRes.json();

if (lock.status === "duplicate") {
  // Don't re-run side effects. Return your cached result (DB/Redis) if you store it.
  return await getCachedResult(key);
}

// New action: execute once, then store result under the same key.
const result = await processOrder({ userId, items: cartItems, total: cartTotal });
await saveCachedResult(key, result);
return result;