Tuning Variance Thresholds to Cut False-Positive Audit Flags

This page fixes the audit queue nobody works any more: a single flat variance tolerance that fires on rounding noise and legitimate fuel swings while quietly waving real leakage through.

The Failure You Are Hitting

You set one global tolerance — say, flag any billed charge that deviates more than 2% from the contracted amount — and shipped it. For a week the queue looked useful. Then it filled with thousands of items an analyst learns to ignore, because most of them are not overcharges at all:

  • A linehaul charge is flagged because the carrier rounds to the cent differently than your Decimal engine, producing a 0.4% variance on a $2,900 haul that is pure arithmetic noise.
  • Every fuel-surcharge line in a batch trips the same 2% because the DOE index moved between quote and invoice — a legitimate, contractually correct swing, not leakage.
  • Meanwhile a 1.1% systematic overbill on detention across an entire carrier sails through under the 2% floor, so the one pattern worth recovering never surfaces.

The observable symptom is a precision collapse: the flag rate climbs, the confirmed-recovery rate per flag falls toward single digits, and the analysts who feed Threshold Tuning & Alerting stop opening the queue. An unworked queue recovers nothing, so a noisy audit is functionally identical to no audit at all.

Root Cause Analysis

The defect is not the tolerance value — it is applying one value across charge types and carriers that have completely different variance physics. A flat threshold assumes a homogeneous population; freight billing is anything but.

  1. Heterogeneous charge types share one number. Linehaul rounding lives in the tenths of a percent. Fuel surcharge legitimately swings several percent with the index. Accessorials are often flat dollar amounts where a small absolute delta is a large percentage. A threshold tuned to catch linehaul leakage buries fuel; a threshold loose enough for fuel misses everything else.
  2. Carriers bill with different precision. One carrier’s system rounds every line to the nearest cent; another carries four decimals; a third applies a blanket minimum charge that distorts the percentage on light shipments. The same 2% means different things per carrier_scac.
  3. No precision/recall calibration. The threshold was picked by intuition, never measured against labelled outcomes. Nobody knows the current false-positive rate, so nobody can say whether tightening or loosening helps.
  4. No hysteresis. A charge sitting near the boundary flaps in and out of the queue as tiny inputs jitter, generating churn that looks like new work every run.
Segmented variance-threshold calibration flow A single mixed variance distribution of linehaul, fuel, and accessorial charges enters at the left, where a flat two-percent line cuts through the middle and mislabels rounding noise and legitimate fuel swings as flags. The stream splits into three per-segment threshold bands: linehaul at a tight zero point four percent, fuel at a wide three percent, and accessorial on an absolute-dollar rule. Each band is calibrated against labelled history to trade precision against recall. Downstream, a hysteresis gate with separate raise and clear levels prevents boundary charges from flapping. Charges above their own segment threshold exit as a FLAG to the analyst queue; charges below exit as PASS. Mixed variance distribution linehaul · fuel · accessorial flat 2% Per-segment thresholds by charge type & carrier linehaul tight 0.4% rounding-safe fuel surcharge wide 3% index-aware accessorial ± $8 absolute flat-fee rule Calibrate on labelled history precision vs recall trade-off pick per segment at target precision maximise recall Hysteresis gate raise ≥ T_high clear ≤ T_low no flapping at the boundary stable queue FLAG analyst queue above segment T PASS auto-approve within tolerance

Reproducible Diagnostic

Do not touch a threshold before you have measured the current false-positive rate against labelled outcomes. If you have historical flags with an analyst disposition (confirmed_recovery vs no_issue), this snippet quantifies precision per segment so you can see exactly where the noise lives:

import polars as pl

# labelled history: one row per prior flag with the analyst's verdict
flags = pl.read_csv("flag_history.csv")  # charge_type, carrier_scac, variance_pct, disposition

summary = (
    flags.with_columns(
        (pl.col("disposition") == "confirmed_recovery").alias("true_positive")
    )
    .group_by(["charge_type", "carrier_scac"])
    .agg(
        pl.len().alias("flags"),
        pl.col("true_positive").sum().alias("recoveries"),
    )
    .with_columns(
        (pl.col("recoveries") / pl.col("flags")).round(3).alias("precision")
    )
    .sort("precision")
)
print(summary)

Read the output as a decision table before you tune anything:

Signal Interpretation Action
precision near 1.0 on a segment threshold is already well placed leave it
precision < 0.2, high flags flat threshold too tight for this charge type widen this segment (Step 2)
Many no_issue at variance_pct just above the line rounding / index noise dominates raise T_high, add hysteresis (Step 3)
Known leakage that never appears as a flag threshold too loose to catch it tighten this segment; recall gap

A single blended precision number hides the problem. Split by charge_type and carrier_scac and the pathology — fuel drowning linehaul, one carrier’s rounding flooding the queue — becomes obvious.

Resolution Path

The fix replaces one global number with a small matrix of thresholds keyed by charge type and carrier, each chosen from labelled history at a target precision, then guarded with hysteresis so items stop flapping. Pin the toolchain so CI and production agree:

# requirements.txt
polars==1.12.0
pydantic==2.10.6
structlog==24.4.0

Step 1 — Segment every charge before you threshold it

Classify each line into a (charge_type, carrier_scac) segment and compute both a percentage and an absolute variance. Accessorials are frequently flat fees, so a percentage alone is misleading — carry the dollar delta too:

from decimal import Decimal

def variance_features(billed: Decimal, contracted: Decimal) -> dict:
    """Both relative and absolute variance — accessorials need the absolute."""
    abs_delta = abs(billed - contracted)
    pct = float(abs_delta / contracted) if contracted else float("inf")
    return {"abs_delta": abs_delta, "variance_pct": pct}

def segment_of(charge_type: str, carrier_scac: str) -> tuple[str, str]:
    # Fall back to a carrier-agnostic bucket when we lack per-carrier history.
    return (charge_type.strip().upper(), carrier_scac.strip().upper())

A common mistake is to threshold fuel and linehaul on the same percentage. Fuel legitimately tracks the DOE index — cross-reference Detecting Fuel-Surcharge Drift Across Invoice Batches before you decide a fuel swing is leakage rather than a correct index move.

Step 2 — Calibrate each segment threshold from labelled history

For each segment, sweep candidate thresholds and pick the tightest one that still meets your target precision, maximising recovered dollars. This is the precision/recall trade-off made explicit rather than guessed:

def calibrate_threshold(rows: list[dict], target_precision: float = 0.7) -> float:
    """
    rows: [{'variance_pct': float, 'true_positive': bool}, ...] for ONE segment.
    Return the lowest threshold whose flagged set still hits target precision,
    so we catch as much leakage as possible without flooding the queue.
    """
    candidates = sorted({r["variance_pct"] for r in rows})
    best = candidates[-1] if candidates else 0.02
    for t in candidates:
        flagged = [r for r in rows if r["variance_pct"] >= t]
        if not flagged:
            continue
        tp = sum(1 for r in flagged if r["true_positive"])
        precision = tp / len(flagged)
        if precision >= target_precision:
            best = t          # tightest threshold meeting precision → best recall
            break
    return round(best, 4)

Store the result as a lookup, not a constant. A segment with too little history falls back to a charge-type default so a brand-new carrier is never left with an undefined threshold.

Step 3 — Apply per-segment thresholds with hysteresis

Evaluate each charge against its own segment threshold, and split that threshold into a raise level and a clear level so a charge hovering on the boundary does not flap in and out of the queue on every run:

class ThresholdBook:
    def __init__(self, table: dict[tuple[str, str], float], hysteresis: float = 0.2):
        self.table = table                 # (charge_type, carrier) -> variance_pct
        self.hysteresis = hysteresis       # clear band as a fraction of the threshold

    def decide(self, seg: tuple[str, str], variance_pct: float,
               abs_delta: Decimal, currently_flagged: bool) -> str:
        t = self.table.get(seg) or self.table.get((seg[0], "*"), 0.02)
        t_high = t
        t_low = t * (1 - self.hysteresis)  # must fall below this to clear
        # Absolute-dollar guard so a flat accessorial fee is judged on $, not %.
        if seg[0] == "ACCESSORIAL" and abs_delta < Decimal("8"):
            return "PASS"
        if currently_flagged:
            return "FLAG" if variance_pct >= t_low else "PASS"
        return "FLAG" if variance_pct >= t_high else "PASS"

Rows that come back FLAG are the ones worth an analyst’s time; they route onward to Charge Variance Classification for typing before dispute. The threshold book decides whether to look, not what kind of variance it is.

Verification

Prove the queue got quieter without going blind. These assertions belong in the suite that runs whenever the threshold book is recalibrated:

from decimal import Decimal

def test_fuel_swing_within_band_does_not_flag():
    book = ThresholdBook({("FUEL", "ABCD"): 0.03})
    # a 2.1% legitimate index move must pass under the wide fuel band
    assert book.decide(("FUEL", "ABCD"), 0.021, Decimal("14"), False) == "PASS"

def test_small_linehaul_rounding_does_not_flag():
    book = ThresholdBook({("LINEHAUL", "ABCD"): 0.004})
    assert book.decide(("LINEHAUL", "ABCD"), 0.0009, Decimal("2.6"), False) == "PASS"

def test_systematic_accessorial_overbill_flags():
    book = ThresholdBook({("ACCESSORIAL", "EXLA"): 0.02})
    # $19 over on a flat detention fee clears the absolute guard and flags
    assert book.decide(("ACCESSORIAL", "EXLA"), 0.11, Decimal("19"), False) == "FLAG"

def test_hysteresis_prevents_flapping():
    book = ThresholdBook({("LINEHAUL", "ABCD"): 0.01}, hysteresis=0.2)
    # already flagged at 0.011; a jitter to 0.009 stays flagged (above t_low=0.008)
    assert book.decide(("LINEHAUL", "ABCD"), 0.009, Decimal("30"), True) == "FLAG"

In production the health signal is the ratio of confirmed recoveries to total flags per segment. Track it per run: if precision on a segment falls two runs running, a carrier changed its billing precision or an index re-pegged — recalibrate that segment, do not blanket-loosen everything.

Preventive Configuration

Encode the policy so it survives the next analyst rotation, and make recalibration a scheduled job rather than a heroic one-off:

variance_thresholds:
  default_by_charge_type:
    LINEHAUL: 0.004        # rounding-tight
    FUEL: 0.03             # index-aware, deliberately wide
    ACCESSORIAL: 0.02
  accessorial_absolute_floor_usd: 8   # ignore sub-$8 flat-fee deltas
  hysteresis_fraction: 0.2            # clear band below the raise level
  calibration:
    target_precision: 0.7
    min_labelled_rows_per_segment: 40 # else fall back to charge-type default
    recalibrate_every_days: 14
  alert_on_precision_drop_runs: 2     # two runs below target → page the owner
  • Never ship a bare global tolerance. CI should reject a config that defines only a single number with no per-charge-type breakdown.
  • Minimum sample gate. A segment with fewer than min_labelled_rows_per_segment labelled outcomes inherits the charge-type default rather than an overfit threshold from three data points.
  • Feed the alert stream. Export per-segment precision to Threshold Tuning & Alerting so a drift alert points at the exact segment, and let Accessorial Charge Scoring weight the flags it receives.

FAQ

Why not just raise the global threshold until the queue is quiet?

Because a single number that silences fuel noise is far too loose for linehaul, where real leakage lives in the tenths of a percent. You would trade a noisy queue for a blind one. Segment the threshold by charge type and carrier so each population is judged on its own variance physics, then tighten linehaul and widen fuel independently.

How do I pick the target precision for calibration?

Work backward from analyst capacity. If a reviewer can work 40 flags a day and you want most of them to be real, a target precision around 0.7 keeps the queue worth opening while still catching leakage. Sweep thresholds against labelled history and pick the tightest one that clears that precision, which maximises recall — recovered dollars — at the precision you can staff.

What is hysteresis doing here and why do I need it?

It stops charges on the boundary from flapping. With one threshold, a line at 1.01% flags, drops to 0.99% next run and clears, then flags again — churn that looks like new work. Splitting into a raise level and a lower clear level means a charge must move meaningfully back inside tolerance before it leaves the queue, so the queue stays stable.

My accessorial flags are all tiny percentages that are still real overbills. What gives?

Accessorials are often flat fees, so a small dollar error is a large percentage and a large dollar error can be a small percentage. Judge them on an absolute-dollar floor as well as a percentage: ignore sub-$8 deltas as noise, but flag a $19 overbill on a flat detention fee even though its percentage looks unremarkable.

Up one level: Threshold Tuning & Alerting · Section: Rule-Based Rate Validation & Accessorial Auditing