Wiring Webhook Endpoints for Real-Time Invoice Intake

This page fixes the three ways a real-time invoice webhook fails under load: it drops bills during a burst, it reprocesses retried deliveries, and it holds the carrier’s connection open past its timeout so the sender retries forever.

The failure you are hitting

You exposed a POST /webhooks/invoices endpoint, the carrier’s system fires it the instant a bill is cut, and in a demo it is beautifully real-time. Under production traffic it degrades in three ways, none of which raises an error you would notice:

  • Your handler parses the invoice, validates it, and writes to the ledger inside the request. When the carrier bursts 400 invoices in a few seconds — end-of-day billing runs do exactly this — each request holds a worker for hundreds of milliseconds, the pool saturates, new POSTs queue behind it, and some are shed with 503. Real invoices are dropped at the front door.
  • The carrier’s webhook client retries on any non-2xx or on a slow response. Because your handler does real work before answering, a request that was actually succeeding but took too long gets retried, and your endpoint processes the same bill two or three times. The same freight charge lands in the ledger repeatedly.
  • The carrier documents a 5-second delivery timeout. Your synchronous handler occasionally takes six seconds under load, so from the sender’s side every one of those is a failure. It retries with exponential backoff, indefinitely, and a single slow bill becomes a self-sustaining retry storm that adds yet more load.

This is the webhook terminal inside Ingestion Transport Protocols. Unlike an SFTP poll you control the timing of, a webhook’s timing is the sender’s, and every one of these failures comes from doing too much before you answer.

Root cause analysis

None of these are framework limits. They are the result of putting work in the request path that has no business being there:

  1. Synchronous processing in the handler. Parsing, validation, and ledger writes are seconds of work in aggregate. Doing them before returning couples your throughput to the slowest downstream dependency, so a slow database turns into shed webhooks at the edge.
  2. No fast acknowledgement. The sender’s contract is “answer 2xx quickly or I retry”. A handler that answers only after finishing the work will, under load, breach the sender’s timeout on requests that were actually fine, manufacturing retries.
  3. No signature verification. Without an HMAC check, the endpoint cannot distinguish a genuine carrier delivery from a replayed or forged POST, and it cannot safely make the handler idempotent because it does not trust the event-id it is deduping on.
  4. No event dedup and no idempotent consumer. At-least-once delivery is guaranteed by the sender’s retries. Without persisting the event-id and making the async consumer idempotent, every retry becomes a duplicate ledger row.
Thin webhook handler: verify, persist, ack 202, process async A carrier POSTs an invoice webhook to a thin HTTP handler. The handler first verifies the HMAC signature over the raw body; an invalid signature returns 401 without any processing. A valid request has its event-id checked against a dedup store: an id already seen returns 200 immediately without re-enqueueing, while a new id has its raw body persisted, is enqueued to the message broker, and the handler returns a fast 202 Accepted to the carrier. Separately, an async worker pool consumes from the broker and runs an idempotent parse-and-ingest, keyed on the same event-id, entirely outside the request path so bursts and retries never block the sender. Carrier POST invoice retries on non-2xx HMAC verify over raw body constant-time 401 · no work bad sig valid event-id dedup seen before? keystore check 200 · already queued retry / dup new id Persist + enqueue raw body → store event → broker return 202 fast Async lane — outside the request path Message broker invoice.intake buffers bursts Idempotent worker parse + ingest keyed on event-id Ledger write exactly once per event-id
The handler only verifies the signature, dedups the event-id, persists, enqueues, and returns 202 in milliseconds; all parsing and ledger work happens in an async worker pool keyed idempotently on the same event-id.

Reproducible diagnostic

Confirm which failure you have. This load probe fires a burst of POSTs — including a deliberate duplicate event-id — and reports latency percentiles and how many duplicates the endpoint accepted as new:

import asyncio
import time
import httpx


async def fire(client, url, event_id, body, secret):
    sig = sign_hmac(body, secret)
    t0 = time.perf_counter()
    r = await client.post(url, content=body,
                          headers={"X-Event-Id": event_id, "X-Signature": sig})
    return (time.perf_counter() - t0) * 1000, r.status_code


async def probe(url, secret, n=400):
    body = b'{"invoice":"INV-9001","carrier":"ABCD","total":"1420.55"}'
    async with httpx.AsyncClient(timeout=10) as client:
        tasks = [fire(client, url, f"evt-{i}", body, secret) for i in range(n)]
        tasks += [fire(client, url, "evt-0", body, secret)]      # duplicate of the first
        results = await asyncio.gather(*tasks)

    lat = sorted(ms for ms, _ in results)
    shed = sum(1 for _, code in results if code >= 500)
    p95 = lat[int(len(lat) * 0.95)]
    print(f"p50={lat[len(lat)//2]:.0f}ms p95={p95:.0f}ms shed_5xx={shed} n={len(results)}")

# asyncio.run(probe("http://localhost:8000/webhooks/invoices", "shared-secret"))

Read the numbers against this table:

Signal Likely failure Fix
p95 in the hundreds of ms or seconds synchronous work in the handler thin handler + async lane (Steps 1, 4)
shed_5xx > 0 under burst worker pool saturated by request-path work move processing off the request path (Step 4)
duplicate evt-0 produces two ledger rows no event-id dedup dedup store + idempotent consumer (Steps 3, 5)
any request accepted without a valid signature no HMAC verification signature check first (Step 2)

A handler whose p95 sits under ~20 ms while a burst is in flight is doing the right amount of work — everything else has been pushed to the async lane.

Resolution path

The fix is a handler that does four cheap things and nothing else, plus an idempotent async consumer. Pin the deps:

# requirements.txt
fastapi==0.115.4
uvicorn==0.32.0
redis==5.2.0
structlog==24.4.0

Step 1 — Keep the handler thin

The handler’s entire job is verify, dedup, persist, enqueue, answer. No parsing, no validation, no ledger write. Everything expensive is a message on the broker for a worker to pick up.

from fastapi import FastAPI, Request, Response, HTTPException

app = FastAPI()


@app.post("/webhooks/invoices")
async def receive(request: Request):
    raw = await request.body()                       # raw bytes for signature + storage
    event_id = request.headers.get("X-Event-Id", "")
    signature = request.headers.get("X-Signature", "")

    if not verify_signature(raw, signature):         # Step 2
        raise HTTPException(status_code=401, detail="invalid signature")
    if not event_id:
        raise HTTPException(status_code=400, detail="missing event id")

    if not await dedup_claim(event_id):              # Step 3: already queued
        return Response(status_code=200)             # idempotent no-op for a retry

    raw_ref = await raw_store.put(event_id, raw)      # persist before enqueue
    await broker.publish("invoice.intake", {         # Step 4: hand off
        "event_id": event_id, "raw_ref": raw_ref, "transport": "webhook",
    })
    return Response(status_code=202)                 # fast ack; work happens async

Common mistake: calling await parse_and_validate(raw) here “because it is fast enough”. It is fast enough at one request per second and catastrophic at 400 in a burst. The handler must stay constant-time regardless of payload complexity; move all variable-cost work behind the broker.

Step 2 — Verify the HMAC signature in constant time

Compute the expected HMAC over the exact raw bytes and compare with a constant-time function so a forged or replayed POST is rejected before any state changes. Constant-time comparison prevents a timing side channel from leaking the secret.

import hashlib
import hmac


def verify_signature(raw: bytes, provided: str, secret: bytes = b"shared-secret") -> bool:
    """Constant-time HMAC-SHA256 check over the raw request body."""
    expected = hmac.new(secret, raw, hashlib.sha256).hexdigest()
    # compare_digest avoids leaking match length via response timing
    return hmac.compare_digest(expected, provided or "")

Common mistake: verifying against a re-serialized body. If you parse the JSON and re-dump it before hashing, key ordering or whitespace differences change the bytes and every signature fails. Always HMAC the raw body exactly as received.

Step 3 — Claim the event-id atomically

Dedup on the sender’s event-id using an atomic set-if-absent so a retry is recognized before it is enqueued. A Redis SET NX with a TTL is ideal: the first delivery wins the claim, retries see the key and short-circuit to a 200.

import redis.asyncio as aioredis

_redis = aioredis.from_url("redis://localhost:6379")


async def dedup_claim(event_id: str, ttl_seconds: int = 7 * 24 * 3600) -> bool:
    """Return True only if this event_id was not already claimed."""
    # SET key value NX EX ttl -> True on first set, None if it already existed
    claimed = await _redis.set(f"wh:evt:{event_id}", "1", nx=True, ex=ttl_seconds)
    return bool(claimed)

Common mistake: a GET then SET instead of SET NX. Under the burst that causes the failure, two concurrent retries both GET a miss and both enqueue. The claim must be a single atomic operation so exactly one caller wins.

Step 4 — Enqueue to the broker and return fast

The 202 Accepted is the whole point: it tells the carrier “received, will process” so its retry timer resets, while the actual work is now a durable message the broker buffers through the burst. Persist raw bytes before publishing so a consumer never races to a missing raw_ref.

class Broker:
    def __init__(self, stream: str = "invoice.intake"):
        self.stream = stream

    async def publish(self, stream: str, event: dict) -> None:
        # XADD onto a Redis Stream (or produce to Kafka) — durable, ordered, buffered.
        await _redis.xadd(stream, {"payload": _json_dumps(event)})

Common mistake: returning 202 before the enqueue durably commits. If you ack then fail to publish, the carrier stops retrying and the invoice is lost. Ack only after the persist and the enqueue have both committed.

Step 5 — Make the async consumer idempotent

The consumer does the real work — parse, validate, hand to the ledger — and must be safe to run twice, because broker redelivery and the occasional duplicate that slips the edge check are both possible. Key the ledger write on the same event-id.

import structlog

logger = structlog.get_logger()


async def consume_once(entry: dict) -> None:
    event_id = entry["event_id"]
    raw = await raw_store.get(entry["raw_ref"])

    # Idempotent guard at the point of persistence, not just at the edge.
    async with ledger.transaction() as tx:
        if await tx.event_already_committed(event_id):
            logger.info("webhook_replay_noop", event_id=event_id)
            return                                   # redelivery: do nothing
        invoice = parse_invoice(raw)                 # the expensive work, off the request path
        await tx.commit_invoice(invoice, source_event_id=event_id)
    logger.info("webhook_ingested", event_id=event_id, invoice=invoice.number)

Common mistake: relying only on the edge dedup and treating the consumer as at-most-once. The edge check reduces duplicates; it does not eliminate broker redelivery. The ledger write itself must be idempotent on event-id, backed by a unique constraint, so the exactly-once guarantee lives where the money is.

Verification

Prove the failures are closed. These run against the app with a fake broker and ledger in CI:

import pytest


@pytest.mark.asyncio
async def test_handler_returns_fast_and_does_not_parse(client, spy_parser):
    r = await client.post("/webhooks/invoices", content=b'{"invoice":"INV-1"}',
                          headers=signed_headers("evt-1", b'{"invoice":"INV-1"}'))
    assert r.status_code == 202
    assert spy_parser.call_count == 0            # no parsing in the request path


@pytest.mark.asyncio
async def test_duplicate_event_id_enqueues_once(client, fake_broker):
    hdrs = signed_headers("evt-2", b'{"invoice":"INV-2"}')
    await client.post("/webhooks/invoices", content=b'{"invoice":"INV-2"}', headers=hdrs)
    r2 = await client.post("/webhooks/invoices", content=b'{"invoice":"INV-2"}', headers=hdrs)
    assert r2.status_code == 200                 # retry short-circuits
    assert fake_broker.count("evt-2") == 1


@pytest.mark.asyncio
async def test_bad_signature_rejected(client):
    r = await client.post("/webhooks/invoices", content=b'{"invoice":"INV-3"}',
                          headers={"X-Event-Id": "evt-3", "X-Signature": "deadbeef"})
    assert r.status_code == 401


@pytest.mark.asyncio
async def test_consumer_is_idempotent(fake_ledger):
    entry = {"event_id": "evt-4", "raw_ref": "mem://evt-4"}
    await consume_once(entry)
    await consume_once(entry)                    # redelivery
    assert fake_ledger.rows_for("evt-4") == 1

The healthy production signal is a p95 handler latency in single-digit milliseconds, a webhook_ingested count that tracks distinct event-ids, and a nonzero-but-flat webhook_replay_noop proving retries are absorbed rather than duplicated. A rising handler latency means work crept back into the request path — profile the handler, do not just add workers.

Preventive configuration

Encode the guarantees so the regression cannot return:

webhook_intake:
  handler_budget_ms: 150          # hard ceiling; alert if p95 approaches the sender timeout
  signature: hmac_sha256_raw_body # verify over raw bytes, constant-time compare
  dedup_key: event_id             # sender's id, atomic SET NX claim
  dedup_ttl_days: 7               # covers the sender's retry-backoff window
  ack_status: 202                 # only after persist + enqueue commit
  process: async_only             # zero parsing/validation in the request path
  consumer_idempotent: true       # ledger write unique on event_id
  • Atomic claim. The edge dedup is a Redis SET NX; the ledger write is UNIQUE(source_event_id). Two layers, both atomic, so neither a burst nor a redelivery duplicates a bill.
  • Alert on handler latency. Page when p95 approaches the carrier’s documented timeout — that is the early warning for a retry storm, well before 5xx shedding starts.
  • Shared canonical key. The event_id enqueued here is the dedup_key Ingestion Transport Protocols keys the intake event on, and the same idempotency discipline appears in Handling Dispute Webhook Callbacks Idempotently on the dispute side.

FAQ

Why return 202 instead of 200 from the webhook?

Because the work has not happened yet — it has only been accepted for processing. A 202 Accepted honestly signals “received, queued, will process”, which is exactly true once the raw body is persisted and the event is enqueued. It resets the sender’s retry timer immediately while the parse and ledger write proceed asynchronously. The status code matters less than the discipline of answering before doing the work.

The carrier keeps re-sending the same invoice. How do I stop double-processing?

Dedup on the sender’s event-id at two layers. At the edge, an atomic Redis SET NX claim short-circuits a retry to a fast 200 before it is enqueued. At the ledger, a unique constraint on the source event-id makes the async write idempotent even if the broker redelivers. The edge check reduces duplicates and the ledger constraint eliminates them.

Can I just verify the signature and parse in the same handler if parsing is quick?

No. Parsing that is quick for one request is fatal for a burst of several hundred arriving in seconds, which is exactly when end-of-day billing fires. Any variable-cost work in the request path couples your edge throughput to your slowest dependency and eventually sheds real invoices. Keep the handler constant-time and move all parsing behind the broker.

Should I HMAC the parsed JSON or the raw body?

The raw body, exactly as received. If you parse and re-serialize before hashing, differences in key ordering or whitespace change the bytes and every signature fails to verify. Read the raw request bytes once, use them both for the HMAC check and for persistence, and only parse later in the async consumer.

Up one level: Ingestion Transport Protocols · Section: Automated Invoice Parsing & EDI/XML Ingestion