Charge Variance Classification

A validation verdict tells you a line is wrong; it does not tell you how it is wrong, and the how decides everything that follows. Charge variance classification is the stage that turns an undifferentiated “this line breached tolerance” flag into a typed, scored case — a linehaul overcharge, a duplicate billing, an accessorial that is not on the contract, fuel-surcharge drift, a weight or zone error, or a reweigh/reclass — and then decides whether that case is worth disputing at all. It sits at the front of Discrepancy Resolution & Dispute Routing: the router downstream cannot pick a carrier channel, assemble an evidence package, or check a filing deadline until this stage has named the variance and attached a confidence it can trust.

The reason to build this as a real engine rather than a stack of if statements is that the variance types overlap. A line can look like both an overcharge and a duplicate; a fuel line can look like a rate error when it is really index drift. A classifier that assigns exactly one type with a calibrated confidence — and abstains to manual review when the signals conflict — is what keeps the wrong claim from reaching the wrong channel. This stage does not adjudicate the dollar amount and it does not file anything; it produces a VarianceCase the dispute router can act on deterministically.

Prerequisites

This stage is a transform: given one flagged line plus its resolved rate context, it emits one typed, scored case. It assumes the upstream tiers have already done their jobs.

The classifier reads, it never re-derives: if weight-and-zone validation says the billed weight is fine, this stage does not second-guess it. Feeding the engine a signal it did not compute is what keeps the taxonomy honest and the confidence meaningful.

Architecture: from flagged line to typed case

The engine is a feature-extraction front end feeding a bank of per-type rules, whose scores are reconciled into a single winning type and a confidence. The hard part is not any one rule — it is arbitrating between types when several fire, which is exactly what the diagram below makes explicit.

The signals map to variance types below. A single line can trip more than one signal, which is precisely why the aggregator scores every type rather than short-circuiting on the first match.

Step-by-step implementation

The engine is four stages: extract features, score each type, reconcile to a single type with confidence, then gate on dispute-worthiness. Keep each stage a pure function so the whole classifier is testable without a database.

Step 1 — Extract normalized features

Derive the signals once, from the flagged line and its rate context, into a flat feature record. Everything downstream reads this record, never the raw inputs — which keeps the type rules from silently disagreeing about what “the delta” is.

# classification/features.py
from decimal import Decimal
from dataclasses import dataclass


@dataclass(frozen=True)
class Features:
    contract_delta: Decimal      # billed - expected, signed
    delta_ratio: float           # delta as fraction of expected
    fingerprint_hit: bool        # same PRO+charge seen in ledger
    not_on_schedule: bool        # accessorial code absent from contract
    fsc_delta: Decimal           # billed FSC - recomputed FSC
    weight_zone_flag: bool       # upstream cross-check failed


def extract(line: dict, rate_ctx: dict, ledger_index) -> Features:
    billed = Decimal(str(line["billed_amount"]))
    expected = Decimal(str(rate_ctx["expected_amount"]))
    delta = billed - expected
    # Guard against a zero expected — an off-contract charge has no
    # contracted baseline, so ratio is undefined, not infinite.
    ratio = float(delta / expected) if expected != 0 else 0.0
    return Features(
        contract_delta=delta,
        delta_ratio=ratio,
        fingerprint_hit=ledger_index.seen(line["pro_number"], line["charge_fingerprint"]),
        not_on_schedule=(line["charge_code"] not in rate_ctx["accessorial_schedule"]),
        fsc_delta=Decimal(str(line.get("billed_fsc", "0"))) - Decimal(str(rate_ctx.get("expected_fsc", "0"))),
        weight_zone_flag=bool(line.get("weight_zone_failed", False)),
    )

Common mistake: computing the delta in float. A sub-cent representation error flips the sign of a near-zero delta and mis-types a clean line as an overcharge. Keep every monetary feature in Decimal and only cast to float for the unitless ratio.

Step 2 — Score each variance type

Give every type a scoring rule that returns a value in [0, 1]. The rules are intentionally independent — none knows the others exist — so adding a new variance type never perturbs the existing ones.

# classification/rules.py
from decimal import Decimal
from models.dispute import VarianceType

OVERCHARGE_TOL = Decimal("0.01")   # 1% band absorbs rounding drift


def score_types(f) -> dict:
    scores = {}

    # Linehaul overcharge: positive delta beyond tolerance, and the charge
    # IS on the contract (otherwise it is an accessorial mismatch instead).
    if f.contract_delta > 0 and not f.not_on_schedule:
        scores[VarianceType.LINEHAUL_OVERCHARGE] = min(1.0, abs(f.delta_ratio) / 0.05)

    # Duplicate billing: a fingerprint hit is near-conclusive on its own.
    if f.fingerprint_hit:
        scores[VarianceType.DUPLICATE_BILLING] = 0.95

    # Accessorial mismatch: charge is simply not on the contracted schedule.
    if f.not_on_schedule and f.contract_delta > 0:
        scores[VarianceType.ACCESSORIAL_MISMATCH] = 0.85

    # Fuel-surcharge drift: billed FSC exceeds the recomputed indexed value.
    if f.fsc_delta > OVERCHARGE_TOL:
        scores[VarianceType.FUEL_SURCHARGE_DRIFT] = min(1.0, float(f.fsc_delta) / 25.0)

    # Weight / zone error: trust the upstream cross-check outright.
    if f.weight_zone_flag:
        scores[VarianceType.WEIGHT_ZONE_ERROR] = 0.90

    return scores

Common mistake: letting overcharge and accessorial-mismatch both fire on an off-contract charge. Gate the overcharge rule on not f.not_on_schedule so a charge with no contracted baseline is typed as a mismatch, not a rate overcharge — the two go to different evidence packages and, often, different channels.

Step 3 — Reconcile to one type with a confidence margin

The winning type is the argmax, but the confidence is the margin between the top score and the runner-up. A top score of 0.9 means little if the second-place type also scored 0.88 — that is an ambiguous line a human should see, not a claim to fire automatically.

# classification/reconcile.py
from dataclasses import dataclass
from models.dispute import VarianceType


@dataclass
class VarianceCase:
    variance_type: VarianceType | None
    confidence: float
    claimed_amount: object


def reconcile(scores: dict, claimed_amount) -> VarianceCase:
    if not scores:
        return VarianceCase(None, 0.0, claimed_amount)   # nothing fired
    ranked = sorted(scores.items(), key=lambda kv: kv[1], reverse=True)
    top_type, top_score = ranked[0]
    runner_up = ranked[1][1] if len(ranked) > 1 else 0.0
    # Confidence blends absolute strength with separation from the runner-up,
    # so two near-tied types yield a deliberately low confidence.
    margin = top_score - runner_up
    confidence = round(0.5 * top_score + 0.5 * margin, 4)
    return VarianceCase(top_type, confidence, claimed_amount)

Common mistake: using the raw top score as confidence. It hides ambiguity — the router then submits a claim on the wrong type with false confidence. The margin term is what makes the abstention in the next step meaningful.

Step 4 — Gate on dispute-worthiness

Finally, decide the disposition. A confident case above the carrier minimum is dispute-worthy; a thin-margin case goes to review; anything below the minimum claim is a write-off. This is the exact contract the router in the parent stage consumes.

# classification/gate.py
from decimal import Decimal


def decide(case, min_claim: Decimal, min_confidence: float = 0.80) -> dict:
    if case.variance_type is None or case.claimed_amount < min_claim:
        return {"disposition": "WRITE_OFF"}
    if case.confidence < min_confidence:
        return {"disposition": "MANUAL_REVIEW", "type": case.variance_type}
    return {
        "disposition": "DISPUTE_WORTHY",
        "type": case.variance_type,
        "confidence": case.confidence,
    }

Validation and testing

Because the classifier is a chain of pure functions, it is cheap to test exhaustively, and the cases that matter are the ambiguous ones — a line that looks like two types at once — not the clean single-signal happy paths.

import pytest
from decimal import Decimal
from classification.features import Features
from classification.rules import score_types
from classification.reconcile import reconcile
from models.dispute import VarianceType


def _features(**kw) -> Features:
    base = dict(contract_delta=Decimal("0"), delta_ratio=0.0, fingerprint_hit=False,
                not_on_schedule=False, fsc_delta=Decimal("0"), weight_zone_flag=False)
    base.update(kw)
    return Features(**base)


def test_clean_overcharge_types_high_confidence():
    f = _features(contract_delta=Decimal("120"), delta_ratio=0.04)
    case = reconcile(score_types(f), Decimal("120"))
    assert case.variance_type == VarianceType.LINEHAUL_OVERCHARGE
    assert case.confidence > 0.4


def test_fingerprint_hit_wins_duplicate():
    f = _features(fingerprint_hit=True, contract_delta=Decimal("80"), delta_ratio=0.02)
    case = reconcile(score_types(f), Decimal("80"))
    assert case.variance_type == VarianceType.DUPLICATE_BILLING


def test_ambiguous_line_yields_low_confidence():
    # Off-contract AND a positive delta: mismatch vs overcharge sit close.
    f = _features(contract_delta=Decimal("50"), delta_ratio=0.9, not_on_schedule=True)
    case = reconcile(score_types(f), Decimal("50"))
    assert case.confidence < 0.6   # margin is thin -> route to review


def test_nothing_fires_returns_none():
    case = reconcile(score_types(_features()), Decimal("0"))
    assert case.variance_type is None

Maintain fixtures for each pairwise ambiguity: overcharge vs duplicate, overcharge vs accessorial mismatch, and a fuel line that looks like a rate error but is really drift. Each should assert a low confidence, because the correct behaviour on ambiguity is abstention, not a confident guess.

Performance and tuning

Feature extraction is the only stage that touches I/O — the ledger fingerprint lookup — so it dominates latency. Cache the prior-settlement index in memory per batch and the scoring stage runs at CPU speed over Decimal math.

A single worker classifies several thousand lines per second with a warm fingerprint index; a cold index that hits the ledger per line is one to two orders of magnitude slower, which is why the per-batch cache is load-bearing.

Failure modes

Integration points

The output is a VarianceCase plus a disposition — a stable contract the dispute router consumes without ever re-reading the raw line. Dispute-worthy cases carry the type, confidence, and claimed amount the router needs to select a channel and check a filing window. Cases typed DUPLICATE_BILLING are recovered by short-pay rather than an API claim, and the identity signal that makes that classification safe is detailed in Classifying Overcharge vs Duplicate Billing in Python. Fuel-surcharge cases whose drift only appears in aggregate come from Detecting Fuel-Surcharge Drift Across Invoice Batches. Every typed case — disputed, reviewed, or written off — is recorded against the invoice in the Reconciliation Ledger Schema so the audit trail is complete regardless of disposition.

In this section

• Classifying Overcharge vs Duplicate Billing in Python — the shipment-identity and charge-fingerprint technique that keeps a legitimate re-bill of the same PRO from being tagged as an overcharge, and a true duplicate from being paid.
• Detecting Fuel-Surcharge Drift Across Invoice Batches — the aggregate drift statistic that catches fuel-surcharge leakage no per-line tolerance can see, because each line is individually within band.

Related

• Discrepancy Resolution & Dispute Routing — the parent stage that routes the typed cases this engine produces.
• Accessorial Charge Scoring — supplies the schedule snapshot that separates an overcharge from an off-contract charge.
• Weight & Zone Cross-Validation — produces the weight/zone flag consumed as a classification signal.
• Fuel Surcharge Formula Implementation — the recomputed indexed FSC value the drift signal compares against.
• Reconciliation Ledger Schema — records every typed case against its invoice for a complete audit trail.

---

Up: Discrepancy Resolution & Dispute Routing