Vora
A delivery identifier and a business action answer different questions
The delivery record answers, “Have we received this event before?” An action record answers, “Have we already performed the work this event requested?”
Keep both where the integration needs them. A provider can send the same event repeatedly. Separately, different events can lead to the same business action. A payment event and a later reconciliation process might both ask to open the same opportunity.
Define the identity using the client's business rule. For example, the hypothetical workflow might allow one opportunity per paid enquiry, while allowing several legitimate enquiries from the same customer. The customer's email address would be a poor duplicate key: it could suppress a later purchase.
Scope your records to the relevant client, provider account and environment. Include the action when one event starts several operations. A receipt for “create opportunity” does not establish that “send welcome email” completed.
Use opaque identifiers, and avoid putting personal data in keys. If a repeated identity arrives with conflicting parameters, investigate the change instead of quietly treating it as the original request. Amazon's Builders' Library explains why explicit caller intent and parameter checks matter when designing idempotent APIs.
Read the delivery contract before choosing retries
Stripe documents repeated deliveries, out-of-order events and automatic retries for up to three days in live mode. Manual resends do not cancel automatic retries. Those are Stripe's rules; another sender can behave differently.
If you place events on Amazon SQS standard queues, the queue can deliver a message more than once too. At-least-once delivery requires the consumer to handle repetition. It does not establish that the entire business workflow will complete exactly once.
For your sender, record the acknowledgement deadline, retry schedule, redelivery mechanism and event ordering rules. Check how it identifies an event and whether manual replay preserves that identity. GitHub, for example, preserves the X-GitHub-Delivery value on a requested redelivery.
An older event may describe a state that has already changed. Decide whether an operation uses the event's historical facts or needs the current resource state. A delayed “enquiry created” event should not reopen an enquiry the client has cancelled.
Acknowledge receipt after you can recover the work
Verify the webhook with the provider's documented signature or authentication mechanism. Then durably record enough information to process it and identify repeats before acknowledging acceptance.
Choose an ingress design that does not leave a gap between storing the event and making it available to a worker. A database-backed inbox that workers read directly is one option. A transactional outbox is another when work must move from a database to a queue. The implementation depends on your storage and processing system.
Keep the request handler short. The acknowledgement tells the sender that your endpoint accepted the delivery; your internal status must separately show whether the client's action completed.
Two workers can see the same event at the same time. Use an atomic claim or equivalent coordination in shared storage. A “check, then insert” sequence without concurrency protection can allow both workers to proceed.
Do not mark the business action complete when a worker merely claims it. Preserve the distinction between received, processing, confirmed and uncertain. Make abandoned claims recoverable without letting a replacement worker blindly repeat an action that may already have happened.
The remote write leaves a failure window
Your local transaction cannot normally include a write to an unrelated CRM. The CRM can accept the action just before your worker crashes, leaving no local completion record.
Use the destination's idempotency facility where the specific endpoint supports it. Keep the same key and parameters when retrying the same request under that contract. A different client action needs a different identity.
Stripe's API v1 idempotency documentation illustrates the limits: once endpoint execution begins, saved responses include failures, and keys can be removed after they are at least 24 hours old. Reusing a pruned key creates a new request. Your own duplicate records therefore need a retention policy covering the replays you permit; a provider's key lifetime is not a permanent completion ledger.
Stripe also warns that a 500 response can leave an indeterminate result and that switching to a new key may repeat side effects. Read the destination's guidance before treating a server error as permission to try a fresh operation.
When no suitable idempotency facility exists, reconcile using a stable request reference or an authoritative result lookup. Absence from a delayed search index is weak evidence. If you cannot establish whether the action happened, hold it for investigation and tell the responsible operator what remains unknown.
Give each failure a next action
Use a decision table for the actual integration. This example is a starting point for review, not a universal HTTP retry policy.
| Observed state | Next action | Record needed |
|---|---|---|
| Same action already confirmed | Reuse its result; continue only unfinished steps | External record ID and completed action |
| Explicit request validation failure | Correct the request through the agreed process | Provider reason and approved correction |
| Rate limit or temporary unavailability | Follow provider retry guidance within your budget | Attempts, deadline and next eligible time |
| Credential refresh cannot recover access | Hold affected work for authorized reconnection | Connection owner and blocked jobs |
| Timeout or uncertain remote result | Reconcile before any unsafe repeat | Original request identity and evidence |
| Retry or age limit reached | Move to a visible review queue | Named owner and recovery options |
Count retries across the SDK, worker and orchestration layer. Independent retry loops can multiply attempts and costs. Decide which layer owns the policy, and set an elapsed-time limit as well as an attempt limit. Keep jobs recoverable when that budget ends.
For access failures, use the OAuth recovery guide. Reconnection still leaves the question of which jobs should resume.
Test the moments between steps
Run controlled cases that stop the worker after a remote action but before its local completion record. Send concurrent duplicates, replay an older event and test a changed payload using an existing identity. Inspect the resulting records and messages alongside the worker's status.
Test again at the edge of the retention policy. What happens when an operator replays a request after the destination has forgotten its idempotency key? Restrict that replay or reconcile it through the longer-lived action record.
When moving a client workflow, carry these records and ownership rules into the cutover. When scoping a first-client deliverable, include the review queue in the work someone agrees to own.
Return to the hypothetical CRM opportunity. Find the saved opportunity first, attach its ID to the original request and inspect the email step separately. Only the unfinished, still-authorized action should move forward.