Idempotent CRM Sync: Why Your Integration Keeps Creating Duplicates

Every integration between a product and a CRM eventually hits the same wall: a request times out, the job retries, and now there are two opportunities where there should be one. The network call actually succeeded — the response just never made it back.

You cannot prevent that. What you can do is make the second attempt harmless.

Idempotency is a key, not a flag

The mistake I see most often is treating idempotency as a boolean the queue driver provides. It is not. It is a key you choose, and it has to be derived from the business identity of the record, not from the job.

// Wrong: a new UUID per attempt means every retry is a new write.
$externalId = Str::uuid();

// Right: derived from the record, stable across every retry.
$externalId = "portal-deal-{$deal->id}";

With a stable key you stop asking "have I sent this?" and start asking the CRM "do you already have this?" — which is the only question with a trustworthy answer. Salesforce gives you external ID fields with upsert semantics for exactly this reason:

$client->upsert('Opportunity', 'Portal_Deal_Id__c', $externalId, $payload);

One call, safe to repeat. No lookup-then-create race between two workers processing the same deal.

Three places this still goes wrong

Composite records. A deal with line items is not one write. If the parent upserts and the child insert fails, the retry re-upserts a parent that is already correct and re-inserts children that are already there. Give the children their own derived keys — portal-deal-{$deal->id}-line-{$line->id} — not autoincrement identity.

Rate limits treated as failures. A 429 is not an error, it is a scheduling instruction. Retrying it immediately with the same backoff as a real failure burns your remaining quota. Route rate-limited messages to a delay queue and let them come back when the window resets:

public function retryUntil(): DateTime
{
    return now()->addHours(6);
}

public function backoff(): array
{
    return [60, 300, 900, 3600];
}

Dead letters without payloads. A dead-letter queue that stores "job X failed" is an alert, not a recovery tool. Store the full payload as it was at the time of the attempt. Six months in, the ability to replay a failed message by hand is worth more than any dashboard.

What to build first

If you are starting a CRM integration today, build these three things before the first field mapping:

1. A derived external ID on every syncable entity.
2. A delay queue for rate limits, separate from the retry path for genuine failures.
3. A dead-letter store that keeps payloads, with a one-command replay.

Everything else — field mapping, transformation rules, conflict resolution — is easier to change later. These three are load-bearing, and retrofitting them means reconciling the duplicates you have already created.