Handling Dispute Webhook Callbacks Idempotently

This page fixes the webhook handler that corrupts dispute state when carriers re-deliver status callbacks: the same approved event arrives twice and applies a credit twice, or a stale open event lands after denied and drags a closed dispute back to open.

The failure you are hitting

You subscribed to a carrier’s dispute-status webhooks so filed claims update in near-real time. The handler reads the event, updates the dispute, and applies any credit. It passes every manual test, because in a test you send each event once, in order. Carriers do not deliver that way. Webhook delivery is at-least-once and unordered, and your handler shows it:

• The carrier re-delivers an approved event — because your first 200 was slow, or their retry timer fired — and your handler applies the credit a second time. The reconciliation ledger now shows twice the recovered amount, and AP posts a credit that does not exist on the remittance.
• A partial event and the earlier pending event arrive out of order after a delivery backlog drains. The pending lands last and overwrites the dispute back to pending, so a resolved claim looks unresolved and gets re-worked by an analyst.
• A denied event closes a dispute, then a duplicate open from earlier in the queue arrives and reopens it. The dispute oscillates, and whichever event happens to land last wins — the outcome is non-deterministic.

Every one of these silently produces wrong money or wrong state, and none raises an error, because from the handler’s point of view each callback is a perfectly valid request.

Root cause analysis

The handler treats each webhook as an authoritative command to be applied verbatim. Three missing defenses turn that assumption into corruption:

1. No event dedup. Nothing records which event_id values have already been processed, so a re-delivered event is applied again — and a credit applied twice is real money booked twice.
2. No monotonic state machine. State transitions are applied unconditionally instead of being guarded by an allowed-transitions rule, so a stale event can move a dispute backwards — denied back to open, partial back to pending.
3. No signature verification. The endpoint accepts any well-formed POST, so it cannot distinguish a genuine carrier callback from a replayed or forged one, and has no trustworthy basis for acting on the payload at all.

Reproducible diagnostic

Before touching the handler, confirm the corruption is duplicate-and-reorder rather than a genuine carrier error. Replay the raw event log for one dispute and watch what a naive apply would do:

# Raw events as the carrier delivered them, in arrival order.
events = [
    {"event_id": "e1", "status": "pending",  "seq": 1},
    {"event_id": "e2", "status": "approved", "seq": 3, "credit": 214.50},
    {"event_id": "e2", "status": "approved", "seq": 3, "credit": 214.50},  # re-delivery
    {"event_id": "e3", "status": "pending",  "seq": 2},                    # arrived late
]

state = "unknown"
applied_credit = 0.0
seen = set()
for ev in events:
    dup = ev["event_id"] in seen
    seen.add(ev["event_id"])
    if not dup:
        applied_credit += ev.get("credit", 0.0)   # naive: apply unconditionally
    state = ev["status"]                            # naive: latest wins
    print(f"{ev['event_id']} status={ev['status']:8s} dup={dup} "
          f"state_now={state} credit_total={applied_credit}")

Read the final two lines against this table:

The tell is the final line reading state_now=pending after an approved had already arrived: the resolved dispute was silently reopened by a stale event.

Resolution path

The fix layers three guards in front of any state change — verify the signature, dedup the event, and allow only forward transitions — then applies the credit in a way that is safe under at-least-once delivery. No extra dependencies beyond the standard library and your web framework are required.

Step 1 — Verify the webhook signature

Carriers sign the raw request body with a shared secret and send the digest in a header. Recompute the HMAC over the exact raw bytes and compare in constant time. Never parse the JSON before verifying — parsing an unverified body is acting on untrusted input.

import hashlib
import hmac


def verify_signature(raw_body: bytes, header_sig: str, secret: str) -> bool:
    """Constant-time HMAC-SHA256 check over the raw request body."""
    expected = hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
    # compare_digest avoids the timing side-channel of a plain == on secrets.
    return hmac.compare_digest(expected, header_sig or "")

A failed check returns 401 and processes nothing. This is also what makes replay attacks harmless in combination with dedup: even a validly signed replay is caught by the next guard.

Step 2 — Deduplicate on a durable event store

Record every processed event_id in a store with a unique constraint, and treat the insert itself as the dedup test: if the insert conflicts, the event was already handled and this delivery is a replay to be acknowledged and ignored.

class EventDedupStore:
    """Backed by a table with a UNIQUE constraint on event_id."""

    def __init__(self, conn):
        self._conn = conn

    def claim(self, event_id: str) -> bool:
        """Return True if this is the first sighting; False if a replay."""
        cur = self._conn.execute(
            # INSERT ... ON CONFLICT DO NOTHING; rowcount 0 means already seen.
            "INSERT INTO processed_webhook_events (event_id, seen_at) "
            "VALUES (?, CURRENT_TIMESTAMP) "
            "ON CONFLICT(event_id) DO NOTHING",
            (event_id,),
        )
        self._conn.commit()
        return cur.rowcount == 1

Using the database’s unique constraint rather than a read-then-write check closes the race where two concurrent deliveries of the same event both pass an in-memory “have I seen it?” test before either records it.

Step 3 — Guard state transitions with a monotonic machine

Model the dispute lifecycle as an explicit set of allowed forward transitions. An event whose target state is not reachable from the current state — because it is stale or out of order — is ignored, not applied. Terminal states accept no further transitions.

# Allowed forward transitions; terminal states map to an empty set.
_ALLOWED = {
    "unknown":   {"submitted", "pending"},
    "submitted": {"pending", "approved", "partial", "denied"},
    "pending":   {"approved", "partial", "denied"},
    "approved":  set(),      # terminal
    "partial":   set(),      # terminal
    "denied":    set(),      # terminal
}


def can_transition(current: str, incoming: str) -> bool:
    """True only for an allowed forward move; blocks stale/backward events."""
    if current == incoming:
        return False                       # same-state replay, nothing to do
    return incoming in _ALLOWED.get(current, set())

The transition table is the single source of truth for what “forward” means, so a pending arriving after approved is rejected by construction — pending is not in approved’s (empty) allowed set — and the resolved dispute cannot regress.

Step 4 — Apply the credit at-least-once safely

Combine the three guards in the handler. Because the same event can be delivered more than once, the credit application must be idempotent on its own key even inside the guarded path, so apply it conditionally on the dedup claim.

def handle_webhook(raw_body: bytes, headers: dict, deps) -> tuple[int, str]:
    if not verify_signature(raw_body, headers.get("X-Carrier-Signature"), deps.secret):
        return 401, "bad signature"

    import json
    event = json.loads(raw_body)          # safe now: signature verified
    event_id = event["event_id"]

    # Dedup first: a replay is acknowledged so the carrier stops retrying.
    if not deps.dedup.claim(event_id):
        return 200, "duplicate ignored"

    dispute = deps.ledger.get(event["dispute_id"])
    incoming = event["status"]
    if not can_transition(dispute.status, incoming):
        # Stale or backward — record that we saw it, change nothing.
        return 200, "stale transition ignored"

    # Verified, novel, forward: apply exactly once.
    deps.ledger.transition(
        dispute_id=dispute.dispute_id,
        new_status=incoming,
        credited_amount=event.get("credit"),
        source_event_id=event_id,          # ties the ledger row to the event
    )
    return 200, "applied"

Returning 200 for duplicates and stale events is deliberate: a non-2xx tells the carrier to keep retrying, which would just re-deliver an event you have correctly chosen to ignore.

Verification

Prove the handler is safe under replay and reordering by feeding it the exact adversarial sequence from the diagnostic and asserting the credit is applied once and the state never regresses:

def test_duplicate_event_credits_once():
    deps = build_test_deps(current_status="pending")
    body = signed_event(deps, event_id="e2", dispute_id="D-1", status="approved", credit=214.50)

    first = handle_webhook(body, _sig_header(deps, body), deps)
    second = handle_webhook(body, _sig_header(deps, body), deps)

    assert first == (200, "applied")
    assert second == (200, "duplicate ignored")
    assert deps.ledger.total_credit("D-1") == 214.50   # not 429.00


def test_stale_event_does_not_reopen_closed_dispute():
    deps = build_test_deps(current_status="denied")
    body = signed_event(deps, event_id="e9", dispute_id="D-1", status="pending")

    status, msg = handle_webhook(body, _sig_header(deps, body), deps)

    assert (status, msg) == (200, "stale transition ignored")
    assert deps.ledger.get("D-1").status == "denied"   # stayed terminal


def test_forged_signature_is_rejected():
    deps = build_test_deps(current_status="pending")
    body = b'{"event_id":"e1","dispute_id":"D-1","status":"approved"}'
    status, _ = handle_webhook(body, {"X-Carrier-Signature": "deadbeef"}, deps)
    assert status == 401

In production the proof is in the ledger: each dispute has exactly one source_event_id per applied transition, total credit per dispute matches the carrier remittance, and no dispute row ever transitions out of a terminal state.

Preventive configuration

Make the guards configuration rather than convention so they cannot be quietly removed:

dispute_webhooks:
  verify_signature: true
  signature_header: X-Carrier-Signature
  signature_algo: hmac-sha256
  dedup:
    store: postgres
    unique_on: event_id
    retain_days: 30                 # keep long enough to outlast carrier retries
  state_machine:
    terminal_states: [approved, partial, denied]
    reject_backward_transitions: true
  ack_duplicates_with: 200          # never make the carrier retry an ignored event
  apply_credit_keyed_on: source_event_id

• Signature-required gate. Refuse to boot the handler if the shared secret is unset; a webhook endpoint that silently accepts unsigned traffic is an open write path into the ledger.
• Dedup retention alarm. Ensure the dedup table’s retain_days exceeds the carrier’s maximum retry window, or a late re-delivery after purge will be treated as novel and double-apply.
• Terminal-state assertion. Add a ledger constraint that blocks any update out of a terminal status, so even a bug in the transition table cannot reopen a settled dispute.

FAQ

Why do I sometimes see a credit applied twice for one dispute?

Because the carrier re-delivered the approved event and your handler applied it both times. Webhook delivery is at-least-once, so a slow acknowledgement or the carrier’s retry timer produces duplicates. Record every event_id in a store with a unique constraint and ignore any event you have already processed, applying the credit only on the first sighting.

A resolved dispute went back to pending — how?

An older, lower-sequence event arrived after the resolving one, and a handler that lets the latest arrival win overwrote the resolved state. Model the lifecycle as a monotonic state machine with an explicit allowed-transitions table and treat terminal states as accepting no further transitions, so a stale pending after approved is ignored by construction.

Should a duplicate or stale webhook return an error status?

No — return 200. A non-2xx response tells the carrier the delivery failed and it should keep retrying, which just re-delivers an event you have deliberately chosen to ignore. Acknowledge duplicates and stale events with 200 and simply change nothing; reserve 401 for a failed signature check and 5xx for a genuine processing fault worth retrying.

Do I need signature verification if the endpoint URL is secret?

Yes. A secret URL is not authentication — it leaks through logs, proxies, and browser history, and it does nothing to stop a replay. Verify an HMAC of the raw request body against a shared secret in constant time, and reject anything that fails before you parse the JSON, so the ledger only ever acts on payloads the carrier actually signed.

Related

• Carrier Dispute API Integration — the parent tier whose per-carrier clients register the webhook subscriptions this handler serves.
• Submitting Freight Claims to Carrier Dispute APIs — the outbound half of the lifecycle that files the claims these callbacks report on.
• Wiring Webhook Endpoints for Real-Time Invoice Intake — the inbound webhook plumbing and signature-verification pattern this handler reuses.
• Reconciliation Ledger Schema — the ledger whose state machine and credit rows every applied transition writes to.

Up one level: Carrier Dispute API Integration · Section: Discrepancy Resolution & Dispute Routing