Webhook Retries Are Silent Killers
(And How to Make Them Safe)

Your system received the webhook. Processed it. Everything looked fine. Then the same webhook came again. No error. No warning. Just duplicate side effects.

This is one of the most common reliability failures in modern systems. And most teams don’t even know it’s happening.

🚨 The Rule Nobody Tells You

Webhooks are not exactly-once. Stripe, GitHub, Slack, Shopify, Twilio — all of them retry.

Why? Because delivery is more important than uniqueness. If the sender doesn’t receive a fast confirmation, it assumes the receiver didn’t get it and tries again.

• Your system sees: A new request.
• The sender sees: Reliability through redundancy.
• You get: Duplicate execution.

💥 What Silent Retries Break

Retries don’t show up as failures; they show up as normal successful calls. That’s why this bug is so hard to catch.

🧠 “But Webhook Providers Send Event IDs”

Yes — but they don’t enforce processing rules. You still have to do this yourself:

if event_id_already_seen:
    ignore()
else:
    process()

Most systems don’t do it consistently across every webhook type. So duplicates slip through.

📡 Why Retries Happen Even When Nothing Is “Wrong”

Retries happen under completely normal conditions:

• Temporary network drops
• Your server is slow for 1 second
• A load balancer resets a connection
• A deploy causes a brief hiccup

This is just the internet being the internet. If your app runs in production, retries are guaranteed.

✅ The Safe Architecture

Webhook handlers must be idempotent. Processing the same webhook multiple times must produce the same result as once.

Instead of Webhook → Your backend, you do:

The layer uses the event ID as the idempotency key, blocks duplicates, and makes retries harmless. (If you need to return a previous result, cache it in your own DB/Redis keyed by the same idempotency key.)

🔧 Example: Making Webhooks Safe

Use OnceOnly /v1/check-lock with the provider’s unique event id (Stripe evt_..., GitHub X-GitHub-Delivery, etc):

POST https://api.onceonly.tech/v1/check-lock
Authorization: Bearer once_live_***
Content-Type: application/json

{
  "key": "{{webhook_event_id}}",
  "ttl": 86400,
  "metadata": { "source": "webhook" }
}

Same webhook delivered 5 times? Same key → status=duplicate → one execution.

🤖 When Webhooks Feed Automation or AI

Imagine this flow: Webhook → Zapier → AI Agent → Your API. Each layer adds its own retries. Without idempotency, systems don’t just duplicate — they amplify duplicates.

🚀 The 5-Minute Fix

Webhook providers guarantee delivery. Your app guarantees logic. But nobody guarantees execution safety during retries. That's the OnceOnly gap.

Add an idempotency layer built for webhook-driven systems. It enforces at-most-once execution per key, safe retries, and no duplicate side effects.