A user clicks “Buy.” One order is expected. Two appear in the database. No bug report. No crash. No stack trace. Just duplicate data.
The TL;DR
Duplicate records are typically caused by client retries following network timeouts or race conditions where two identical requests are processed in parallel. To stop this, you must implement an idempotency layer that uses a unique key to ensure any given action is executed exactly once, regardless of how many times it is retried.
This is one of the most common production issues in modern APIs — and it almost never comes from a simple coding mistake.
🚨 The Real Cause: Distributed Systems, Not Bad Logic
Your API doesn’t run in isolation. It sits behind browsers, apps, automation tools, webhooks, load balancers, and networks. Any of these can cause a request to be sent more than once.
Why? Because the client might not know whether the first request succeeded. So it retries. This is the heart of why exactly-once processing is a myth in distributed environments.
🔁 How Duplicates Happen
- Client sends request to create record.
- Your API processes it successfully.
- The network response is delayed or lost (ACK loss).
- Client assumes failure and retries.
Now you have two records. Both requests were valid — your system just wasn’t built for the reality of retries. This is particularly dangerous for payment processing where a retry equals a double charge.
❌ Common Fixes That Fail
| Fix | Why it fails |
|---|---|
| DB uniqueness constraint | Prevents DB insert, but doesn't stop external side effects (like charging Stripe or sending duplicate emails). |
| Check then insert | Race conditions bypass this easily when two threads check at the same time. |
| Disable client retries | Causes data loss when the network actually fails for good. |
✅ The Correct Pattern: Idempotent APIs
APIs that create or modify data must be idempotent. This means the same request can be sent multiple times but results in exactly one successful execution.
Instead of Client → Your API, you do:
The layer requires an idempotency key, stores the result of the first execution, and blocks duplicates. This is how you handle webhook retries and parallel API calls safely.
🔧 Example
POST /orders
Idempotency-Key: order_98765
{
"user_id": "u_123",
"items": [...]
}If the client retries: Same key → same stored result → only one order created.
❓ Frequently Asked Questions
Can't I just use a unique database index?
A unique index prevents the duplicate record, but it doesn't solve the 500 error returned to the user on the second attempt. More importantly, if your API triggers an external action (like an AI tool call or a payment), the index won't stop that action from happening twice before the DB error is even thrown.
How do I generate a good Idempotency Key?
On the client side, use a UUID v4. For webhooks, use the unique event ID provided by the source (e.g., Stripe's `evt_...`). Never derive the key solely from the payload if the user might intentionally want to perform the same action twice (like buying two separate coffees).
Does this slow down my API?
Any idempotency layer adds some latency. With OnceOnly it’s one fast network call (plus a Redis lock on the server side), so it’s typically tens to hundreds of milliseconds depending on region — usually negligible compared to the side effects you’re protecting (payments, emails, external APIs).
How does this apply to AI Agents?
AI agents often "hallucinate" retries or trigger tool calls twice due to context window limits. Implementing idempotency ensures that AI agents don't repeat destructive actions like sending duplicate messages or updating the same record twice.
🚀 The 5-Minute Fix
Add an idempotency layer designed for API-driven systems. It enforces at-most-once side effects per idempotency key, safe retries, and no duplicate side effects.
In distributed systems, a request being sent twice is normal. Processing it twice is optional.
Stop duplicate records from haunting your database.