Submitting Freight Claims to Carrier Dispute APIs

This page fixes the batch-submission failure where a run of freight claims completes partway: some claims come back 409 as duplicates, some 422 on schema, the access token expires mid-run, and without idempotent submission you either double-file the same variance or silently drop it.

The failure you are hitting

You have a queue of classified disputes and a loop that POSTs each one to the carrier’s claims API. It works in a demo with three claims. In production, against a batch of eight hundred, it fails in ways that never raise a clean exception at the top of the loop:

• Roughly a quarter of the way through, the OAuth2 access token crosses its expiry and every subsequent POST returns 401. Your loop either aborts — leaving six hundred claims unfiled — or, worse, keeps going and logs six hundred failures while the disputes sit in limbo.
• A handful of claims return 409 Conflict because a previous crashed run already filed them. Your handler treats 409 as failure, marks them unsubmitted, and re-files them under a fresh reference on the next run — now the carrier has two claims for one variance and credits neither cleanly.
• A few return 422 Unprocessable Entity because one carrier tightened its schema and now rejects a variance_type you never mapped. Your loop retries them five times each, burning rate budget, then drops them without recording why.

The net result is a submission run whose outcome is non-deterministic: re-running it produces a different set of filed, duplicated, and dropped claims each time. That is unacceptable for a process that moves money.

Root cause analysis

None of these are carrier bugs. They are three missing properties in the submitter, each of which a small happy-path batch never exercises:

1. Non-idempotent POST. The POST /claims call has no client-supplied idempotency key, so the carrier cannot tell a genuine resubmission from a brand-new claim. Any retry — after a timeout, a crash, or a 5xx — risks filing a second claim for the same variance.
2. No client-side dedup key. Without a stable key derived from the variance itself, there is nothing to correlate a 409 response back to the claim that already exists, so a conflict looks like an error instead of an idempotent hit.
3. Token lifecycle mishandled. The access token is fetched once at the start of the batch and never refreshed. Long batches outlive the token’s TTL, so every request after the expiry boundary fails 401 even though nothing is actually wrong.

Reproducible diagnostic

Before changing the submitter, confirm which of the three failures dominates your batch. This snippet replays the last run’s response codes from the log and classifies them, so you know whether you are bleeding on tokens, duplicates, or schema:

import collections

# Each line: dispute_id, http_status recorded by the previous submit run.
rows = [
    ("D-1001", 201), ("D-1002", 201), ("D-1003", 401), ("D-1004", 401),
    ("D-1005", 409), ("D-1006", 422), ("D-1007", 401), ("D-1008", 409),
]

buckets = collections.Counter()
for _dispute_id, status in rows:
    if status in (200, 201):
        buckets["ok"] += 1
    elif status == 401:
        buckets["token_expired"] += 1
    elif status == 409:
        buckets["duplicate"] += 1
    elif status in (400, 422):
        buckets["schema"] += 1
    else:
        buckets["other"] += 1

total = len(rows)
for label, n in buckets.most_common():
    print(f"{label:14s} {n:3d}  {n / total:.0%}")

Read the counts against this table to pick where to start:

A token_expired count that rises sharply partway through — rather than being scattered — is the signature of a single expiry boundary, not intermittent auth trouble.

Resolution path

The fix is a submitter that stamps a deterministic idempotency key on every claim, refreshes the token on 401 and replays, classifies each HTTP outcome into retry / recover / dead-letter, and persists the resulting claim_id so a re-run never re-files. Pin the dependencies first:

# requirements.txt
httpx==0.28.1
tenacity==9.0.0

Step 1 — Derive a deterministic idempotency key

The key must be a pure function of the variance, so every retry and every re-run of the same dispute produces the identical key. Never derive it from a timestamp, a UUID minted at send time, or a database auto-increment — any of those changes on retry and defeats the carrier’s dedup.

import hashlib


def idempotency_key(carrier_scac: str, invoice_number: str, variance_type: str) -> str:
    """Stable per-variance key; identical across every retry and re-run."""
    raw = f"{carrier_scac.strip().upper()}|{invoice_number.strip()}|{variance_type.strip()}"
    return hashlib.sha256(raw.encode()).hexdigest()

Send it as the header the carrier documents — commonly Idempotency-Key — so the carrier collapses a resubmission onto the original claim and answers 409 instead of creating a second one. That 409 is now a recoverable signal, because the same key lets you look the existing claim up.

Step 2 — Refresh the token on 401 and replay

Wrap the bearer token in a small manager that refreshes lazily and, critically, forces a refresh when a request comes back 401. The submit path retries the single failed request with the fresh token — it never restarts the batch.

import time
import httpx


class OAuthTokenManager:
    def __init__(self, http: httpx.Client, token_url: str, client_id: str, secret: str):
        self._http = http
        self._token_url = token_url
        self._client_id = client_id
        self._secret = secret
        self._token: str | None = None
        self._expires_at: float = 0.0

    def token(self, force: bool = False) -> str:
        # Refresh 30s early to avoid racing the expiry boundary mid-flight.
        if force or self._token is None or time.monotonic() >= self._expires_at - 30:
            resp = self._http.post(self._token_url, data={
                "grant_type": "client_credentials",
                "client_id": self._client_id,
                "client_secret": self._secret,
            }, timeout=15.0)
            resp.raise_for_status()
            body = resp.json()
            self._token = body["access_token"]
            self._expires_at = time.monotonic() + int(body["expires_in"])
        return self._token


def submit_once(http: httpx.Client, url: str, body: dict, key: str,
                tokens: OAuthTokenManager) -> httpx.Response:
    """POST once; on 401 refresh the token and replay exactly one more time."""
    headers = {"Authorization": f"Bearer {tokens.token()}", "Idempotency-Key": key}
    resp = http.post(url, json=body, headers=headers, timeout=15.0)
    if resp.status_code == 401:
        headers["Authorization"] = f"Bearer {tokens.token(force=True)}"
        resp = http.post(url, json=body, headers=headers, timeout=15.0)
    return resp

Because the idempotency key is stamped before the first attempt, replaying after a 401 cannot double-file: the carrier sees the same key twice and returns the already-created claim on the second call if the first actually landed.

Step 3 — Classify the HTTP outcome and route it

Each response falls into exactly one of four buckets, and conflating any two of them is what causes double-filing or silent drops. Retry transient failures, recover duplicates, dead-letter schema errors, and persist success.

import structlog

log = structlog.get_logger()


class SchemaRejected(Exception):
    """422/400 — do not retry; the payload is wrong, not the timing."""


def handle_response(resp: httpx.Response, dispute_id: str) -> dict:
    code = resp.status_code
    if code in (200, 201):
        return {"dispute_id": dispute_id, "status": "SUBMITTED",
                "claim_id": resp.json()["claimId"]}
    if code == 409:
        # Already filed — the caller recovers the existing id (Step 4).
        return {"dispute_id": dispute_id, "status": "DUPLICATE", "claim_id": None}
    if code in (400, 422):
        log.error("claim_schema_rejected", dispute_id=dispute_id, body=resp.text[:200])
        raise SchemaRejected(f"{code}: {resp.text[:200]}")
    if code == 429 or code >= 500:
        # Transient — surface so the retry/backoff wrapper re-attempts.
        resp.raise_for_status()
    resp.raise_for_status()
    return {}

The 422 deliberately raises rather than returns: it must not be retried, and it must not be recorded as submitted. Its payload goes to a dead-letter store where the field map or carrier schema can be corrected, then the single claim is replayed — no re-running the whole batch.

Step 4 — Persist claim ids and recover duplicates

Persisting the returned claim_id keyed by dispute_id is what makes the next run idempotent even if the carrier ever forgets a key. When a 409 comes back, look up the existing claim by the external reference you sent and store its id so the dispute is correctly marked filed, not failed.

def submit_batch(http, url, tokens, disputes, store) -> dict:
    """Submit a batch idempotently; return a per-outcome tally."""
    tally = {"submitted": 0, "duplicate": 0, "dead_letter": 0}
    for d in disputes:
        if store.get_claim_id(d.dispute_id):
            continue                       # already filed in a prior run — skip
        key = idempotency_key(d.carrier_scac, d.invoice_number, d.variance_type)
        body = build_claim(d)              # carrier-specific mapping
        try:
            resp = submit_once(http, url, body, key, tokens)
            result = handle_response(resp, d.dispute_id)
        except SchemaRejected as exc:
            store.dead_letter(d.dispute_id, reason=str(exc))
            tally["dead_letter"] += 1
            continue
        if result["status"] == "DUPLICATE":
            claim_id = recover_claim_id(http, url, key, tokens)
            store.save_claim_id(d.dispute_id, claim_id)
            tally["duplicate"] += 1
        else:
            store.save_claim_id(d.dispute_id, result["claim_id"])
            tally["submitted"] += 1
    return tally


def recover_claim_id(http, url, key, tokens) -> str:
    headers = {"Authorization": f"Bearer {tokens.token()}"}
    resp = http.get(url, params={"externalReference": key}, headers=headers, timeout=15.0)
    return resp.json()["items"][0]["claimId"]

The upfront store.get_claim_id check is the durable guard: even if the process crashes and restarts, an already-filed dispute is skipped, so the batch is safe to re-run end to end.

Verification

Prove the submitter is idempotent by running the same batch twice against a mock transport and asserting the second run files nothing new:

import httpx


def test_resubmitting_batch_files_no_duplicates():
    calls = {"post": 0}

    def handler(request: httpx.Request) -> httpx.Response:
        if request.method == "POST":
            calls["post"] += 1
            return httpx.Response(201, json={"claimId": f"C-{calls['post']}"})
        return httpx.Response(200, json={"items": [{"claimId": "C-1"}]})

    http = httpx.Client(transport=httpx.MockTransport(handler))
    store = InMemoryStore()
    tokens = _stub_tokens()
    disputes = load_fixture("batch_10.json")

    first = submit_batch(http, "https://api.test/claims", tokens, disputes, store)
    second = submit_batch(http, "https://api.test/claims", tokens, disputes, store)

    assert first["submitted"] == 10
    assert second["submitted"] == 0        # everything already filed
    assert calls["post"] == 10             # no second POST for any dispute


def test_401_then_success_files_one_claim():
    seq = iter([401, 201])

    def handler(request: httpx.Request) -> httpx.Response:
        return httpx.Response(next(seq), json={"claimId": "C-9"})

    http = httpx.Client(transport=httpx.MockTransport(handler))
    resp = submit_once(http, "https://api.test/claims", {}, "k1", _stub_tokens())
    assert resp.status_code == 201

In production the proof is in the tally: a healthy re-run reports submitted=0, and the dead_letter count matches the number of genuinely malformed claims — not a moving number that changes every run.

Preventive configuration

Encode the guards so the regression cannot return silently:

claim_submission:
  idempotency_key: [carrier_scac, invoice_number, variance_type]
  send_idempotency_header: true
  token_refresh_margin_s: 30        # refresh before the expiry boundary
  refresh_on_401: true
  retry_on: [429, 500, 502, 503, 504]
  never_retry_on: [400, 409, 422]   # recover 409, dead-letter 422 — never loop
  persist_claim_id: true            # skip already-filed disputes on re-run
  dead_letter_on_schema_error: true

• Idempotency-key CI check. Assert the key is byte-identical across two builds of the same dispute; a key that changes between runs is a latent double-file waiting for a retry.
• Token TTL guard. Alert if a batch’s runtime exceeds the token TTL and refresh_on_401 is off — that combination guarantees a late-batch 401 cliff.
• Dead-letter alarm. Alert when the dead-letter rate for one carrier rises, because that usually means the carrier changed its claim schema, not that your data went bad.

FAQ

Why do I get 409 Conflict when I re-run a batch that partly failed?

Because the carrier already has those claims from the prior run and your idempotency key matched. That is the system working as intended — a 409 on a stable key means “already filed”, not “error”. Recover the existing claim_id by looking it up on the external reference you sent, mark the dispute filed, and move on. Only a 409 without a corresponding stored claim needs investigation.

Should I fetch a new access token before every request?

No — that adds a full round trip per claim and hammers the token endpoint. Cache the token, refresh it a little before its expires_in boundary, and additionally force a refresh the moment any request returns 401. That covers both the predictable expiry and the case where the carrier revokes early, without a token call on the hot path.

Can I just retry a 422 a few times in case it is transient?

No. A 422 means the payload does not satisfy the carrier’s schema, which is a deterministic property of the request body — it will fail identically every time and only burns your rate budget. Route it to a dead-letter store with the response body, fix the field mapping or the unmapped variance type, then replay that single claim.

What makes a good idempotency key for a freight claim?

A hash of the fields that uniquely identify the disputed movement and reason — carrier SCAC, invoice number, and variance type — never a timestamp, a random UUID minted at send time, or a row id. The key must be reproducible from the dispute alone so that a retry, a crash-restart, or a full batch re-run all generate the same value and the carrier can collapse them onto one claim.

Related

• Carrier Dispute API Integration — the parent tier that defines the adapter registry and per-carrier clients this submitter runs inside.
• Handling Dispute Webhook Callbacks Idempotently — the inbound half of the lifecycle, keeping a filed claim’s status current as the carrier re-delivers events.
• Reconciliation Ledger Schema — where each stored claim_id and its outcome become an auditable ledger event.
• Charge Variance Classification — produces the classified disputes this page submits.

Up one level: Carrier Dispute API Integration · Section: Discrepancy Resolution & Dispute Routing