Detecting Fuel-Surcharge Drift Across Invoice Batches

Fuel surcharge billed per invoice can drift a few cents above the contracted DOE-indexed value on every line while each line stays inside tolerance, so per-line checks pass and the aggregate leaks — this page detects that drift across a whole batch.

The failure you are hitting

Your validation tier recomputes the expected fuel surcharge for each line and flags anything outside a tolerance band. It catches gross errors. What it never catches is the slow bleed: a carrier whose FSC runs 1.2% high on line after line, where 1.2% is comfortably inside your 2% per-line band. No single line trips. But a batch of 40,000 lines at 1.2% over on an average $180 fuel charge is roughly $86,000 of leakage a month that your audit reports as clean.

The tell is a mismatch between the per-line verdict and the reconciliation. Every line passes; the monthly true-up against the carrier’s own published FSC schedule shows a consistent overpayment; and nobody can point to a line to dispute because, individually, none is wrong enough. The variance is real, it is systematic, and it is invisible to a detector that only ever looks at one line at a time.

This is a classification gap inside Charge Variance Classification: the FUEL_SURCHARGE_DRIFT type exists precisely because the signal lives in the aggregate, not the line.

Root cause analysis

Per-line tolerance is the wrong instrument for a systematic bias. Three production conditions produce a drift that hides under it:

  1. Index-version lag. The carrier pegs FSC to a DOE diesel index that updates weekly, but their billing system applies last week’s index value for the first day or two of the new week. Every line in that window is slightly high against the correct peg — a small, one-directional error repeated thousands of times.
  2. Wrong peg week. The contract specifies the index published on the Monday preceding the ship date; the carrier uses the Monday of the ship week. On weeks where diesel rose, that mis-peg overcharges every line by the week-over-week delta.
  3. Rounding accumulation. The carrier rounds the per-mile FSC up at more decimal places than the contract allows, or rounds the surcharge up to the next cent per line. Each rounding is sub-tolerance; summed across a batch, it is a stable positive bias.

All three share a signature: the mean recomputed drift across the batch is positive and tight, not noisy. That is what the detector below measures — the batch statistic, tested against the expected index curve, rather than any single line.

Fuel-surcharge drift detection across an invoice batch An invoice batch on the left feeds a per-line fuel-surcharge recompute stage that re-derives the expected surcharge for every line from the contracted DOE index peg. The per-line drifts flow into a drift-aggregation stage shown as a control chart: a horizontal centerline at zero drift, an upper control limit above it, and a run of plotted points that start near the centerline and then step up and cross the upper control limit. When the batch mean drift breaches the limit, an alert fires to the right, opening a fuel-surcharge-drift case. A note marks that each individual point sits within per-line tolerance even though the aggregate breaches. Invoice batch 40k FSC lines each in tolerance Per-line FSC recompute contracted DOE peg Drift aggregation batch mean vs index curve 0 center UCL batches over time → Alert FUEL_SURCHARGE_DRIFT open case every point is within per-line tolerance — only the aggregate breaches
Each line's fuel surcharge is recomputed against the contracted DOE peg, and the per-line drifts are aggregated as a control chart. Individual points stay inside per-line tolerance, but the batch mean steps up and crosses the upper control limit — the breach that fires a fuel-surcharge-drift case no single-line check could produce.

Reproducible diagnostic

Confirm the drift is systematic, not noise, before building anything. Recompute the expected FSC per line, take the signed drift, and look at its distribution across the batch:

import pandas as pd

lines = pd.read_parquet("fsc_lines.parquet")

# expected_fsc is the contracted recompute; billed_fsc is what the carrier charged.
lines["drift"] = lines["billed_fsc"] - lines["expected_fsc"]
lines["drift_pct"] = lines["drift"] / lines["expected_fsc"]

print(lines["drift_pct"].describe(percentiles=[0.5, 0.9, 0.99]))
print("share of lines outside 2% per-line band:",
      (lines["drift_pct"].abs() > 0.02).mean())
print("batch mean drift $:", lines["drift"].sum().round(2))

Read the shape, not the extremes:

Signal Meaning Verdict
Mean drift_pct ≈ 0, wide spread Random rounding both ways No systematic leak
Mean drift_pct positive and small, tight spread One-directional bias Systematic drift — investigate peg
Share outside 2% band ≈ 0 but batch mean drift large Sub-tolerance leak Per-line check is blind here
Drift steps up mid-batch Index-version lag at week boundary Wrong peg week

A positive mean with a tight spread and almost nothing outside the per-line band is the exact fingerprint of drift: the per-line detector is working as designed and still missing the money.

Resolution path

The fix is a vectorized recompute against the correct index peg, a drift statistic with a significance test, and a control chart that fires a case when the batch mean breaches its limit.

Step 1 — Recompute FSC per line against the correct peg

Vectorize the recompute so a 40k-line batch is a single pass. The peg is the DOE index published on the contract-specified reference day relative to each line’s ship date — the same recompute the validation tier uses, sourced from Fuel Surcharge Formula Implementation.

# drift/recompute.py
import pandas as pd
import numpy as np


def recompute_fsc(lines: pd.DataFrame, index_curve: pd.DataFrame) -> pd.DataFrame:
    """Vectorized per-line expected FSC from the contracted DOE peg."""
    # Resolve each line's peg date, then as-of join the index value in force.
    lines = lines.sort_values("peg_date")
    curve = index_curve.sort_values("index_date")
    joined = pd.merge_asof(
        lines, curve,
        left_on="peg_date", right_on="index_date",
        direction="backward",          # last published value on/before the peg
    )
    # expected_fsc = base linehaul * ((index - floor) / step) * pct_per_step
    joined["expected_fsc"] = (
        joined["linehaul"]
        * np.maximum(0.0, (joined["doe_index"] - joined["fsc_floor"]) / joined["fsc_step"])
        * joined["pct_per_step"]
    ).round(2)
    joined["drift"] = joined["billed_fsc"] - joined["expected_fsc"]
    return joined

Edge case: an as-of join in the wrong direction pegs to next week’s index, which manufactures a phantom drift of its own. direction="backward" enforces “last value published on or before the peg date,” matching the contract wording.

Step 2 — Compute a drift statistic with a significance test

A positive mean is not enough — it must be significant relative to the batch’s own noise. A one-sample test of the mean drift against zero separates a real bias from random rounding.

# drift/statistic.py
import numpy as np
from dataclasses import dataclass


@dataclass
class DriftResult:
    n: int
    mean_drift: float
    total_drift: float
    t_stat: float
    significant: bool


def drift_statistic(drift: np.ndarray, alpha_t: float = 3.0) -> DriftResult:
    n = drift.size
    mean = float(drift.mean())
    sd = float(drift.std(ddof=1)) or 1e-9
    # Standardized mean drift: how many standard errors from zero.
    t = mean / (sd / np.sqrt(n))
    return DriftResult(
        n=n, mean_drift=round(mean, 4), total_drift=round(float(drift.sum()), 2),
        t_stat=round(t, 2),
        significant=(t > alpha_t and mean > 0),   # one-directional, positive only
    )

Common mistake: flagging on total dollars alone. A large batch will always sum to a big number; the standardized statistic is what tells a genuine bias from the noise of scale.

Step 3 — Run a control chart across batches and open a case

Track the per-batch mean drift on a control chart. When the mean crosses the upper control limit — derived from the historical baseline, not a hand-picked constant — open a FUEL_SURCHARGE_DRIFT case for the affected carrier and peg window.

# drift/control_chart.py
from dataclasses import dataclass


@dataclass
class ControlLimits:
    center: float
    ucl: float          # center + 3 sigma of the baseline batch means


def build_limits(baseline_means: list[float]) -> ControlLimits:
    n = len(baseline_means)
    center = sum(baseline_means) / n
    var = sum((m - center) ** 2 for m in baseline_means) / max(1, n - 1)
    sigma = var ** 0.5
    return ControlLimits(center=round(center, 4), ucl=round(center + 3 * sigma, 4))


def evaluate_batch(result, limits: ControlLimits, carrier_scac: str) -> dict | None:
    if result.significant and result.mean_drift > limits.ucl:
        return {
            "variance_type": "FUEL_SURCHARGE_DRIFT",
            "carrier_scac": carrier_scac,
            "claimed_amount": abs(result.total_drift),
            "evidence": {"mean_drift": result.mean_drift, "t_stat": result.t_stat,
                         "ucl": limits.ucl, "lines": result.n},
        }
    return None   # within control — no case

The alert routing and the choice of control limit are governed by Threshold Tuning & Alerting so a drift alert pages the carrier owner rather than firing on normal week-to-week index movement.

Verification

Prove the detector catches a sub-tolerance bias and ignores symmetric noise, using synthetic batches with a known injected drift.

import numpy as np
from drift.statistic import drift_statistic


def test_systematic_drift_is_significant():
    rng = np.random.default_rng(7)
    # 1.2% bias on ~$180 charges: ~$2.16/line, well inside a 2% band.
    drift = rng.normal(loc=2.16, scale=1.0, size=40_000)
    res = drift_statistic(drift)
    assert res.significant
    assert res.mean_drift > 0


def test_symmetric_rounding_is_not_flagged():
    rng = np.random.default_rng(11)
    drift = rng.normal(loc=0.0, scale=1.5, size=40_000)   # noise, no bias
    res = drift_statistic(drift)
    assert not res.significant


def test_small_batch_needs_stronger_signal():
    rng = np.random.default_rng(3)
    drift = rng.normal(loc=1.0, scale=3.0, size=20)       # too few lines
    res = drift_statistic(drift)
    assert not res.significant   # low n -> low t -> no false alarm

In production the proof is the control chart: baseline batches hover near the centerline, and a real drift shows as a sustained run above it, not a single spike. The line-level detail behind a flagged batch — which peg window and which lines contributed — comes straight from the per-line recompute in Calculating Dynamic Fuel Surcharges with Python Formulas.

Preventive configuration

Encode the peg rules and control limits so drift detection is not a one-off script:

fsc_drift:
  peg_reference: monday_preceding_ship_date   # contract-specified peg day
  asof_direction: backward                    # never peg to a future index
  per_line_tolerance_pct: 0.02                # the band that stays blind to drift
  significance_t: 3.0                          # standardized-mean threshold
  control_limit_sigma: 3                       # UCL from baseline batch means
  min_batch_lines: 500                         # below this, do not evaluate
  baseline_window_batches: 30
  • As-of direction gate. Fail CI if the recompute join is ever set to forward — that single flag manufactures phantom drift.
  • Baseline refresh. Recompute control limits on a rolling window so a permanent index regime shift re-baselines instead of alerting forever.
  • Minimum batch size. Skip drift evaluation below min_batch_lines; a small batch cannot produce a trustworthy statistic and only generates false alarms.

FAQ

Why does per-line tolerance miss fuel-surcharge drift entirely?

Because the error is smaller than the band on every line. A 1.2% overcharge sits comfortably inside a 2% per-line tolerance, so no line is ever flagged. The leak only becomes visible when you sum a one-directional bias across thousands of lines — which is exactly what a batch-level drift statistic measures and a per-line check cannot.

How do I tell drift from normal week-to-week diesel movement?

Recompute against the contracted peg, not the raw index. If your recompute already applies the correct DOE value for each line’s ship date, then normal diesel movement cancels out and the remaining mean drift reflects the carrier’s pegging or rounding error, not the market. A significance test on that residual mean separates bias from noise.

Should the drift claim be the total dollars or the per-line mean?

Claim the total dollars of the significant drift, but qualify the case with the standardized statistic. The total is what you recover; the t-statistic and control-limit breach are the evidence that the total is a systematic bias rather than the noise a large batch inevitably sums to. Carriers dispute a bare dollar figure far more easily than a documented control-chart breach.

What causes the drift to appear only at the start of each week?

Index-version lag. The carrier’s billing system carries last week’s DOE value into the first day or two of the new week before refreshing, so every line pegged in that window is high by the week-over-week index delta. It shows as a step up in the control chart at the week boundary and resolves once their system catches up — a signature that points straight at the peg configuration.

Up one level: Charge Variance Classification · Section: Discrepancy Resolution & Dispute Routing