Normalizing Carrier Accessorial Codes with Fuzzy Matching

This page fixes the accessorial charge that validates against nothing because the carrier spelled “detention” in a way your exact-match dictionary has never seen.

The Failure You Are Hitting

Your taxonomy maps carrier accessorial codes to a canonical schedule with an exact dictionary lookup. It works for the codes you have already seen and silently fails for everything else, because the same charge arrives under dozens of carrier-specific spellings and free-text labels:

• One carrier bills DET, another DETENT, a third writes Detention - Loading in a free-text description field, and a fourth sends DTN-01. Your dictionary knows only DETENTION, so three of the four miss.
• A missed code does not raise an error. The charge falls through to an UNMAPPED bucket, never reaches the contracted schedule, and validates against nothing — so a $120 detention fee is neither confirmed nor disputed. It is just paid.
• The UNMAPPED rate climbs every time you onboard a carrier, and an analyst hand-maps the same variants over and over because the mappings live in someone’s head, not a dictionary.

The observable symptom is a growing pile of charges that never get audited. Every unmatched accessorial is unrecovered leakage, and because exact match fails quietly, the pile grows without anyone deciding it should.

Root Cause Analysis

The failure is not the dictionary — it is expecting exact equality across a space that is inherently noisy. Carriers do not share a controlled vocabulary for accessorials, so the input is idiosyncratic by nature.

1. Free-text and idiosyncratic codes. Accessorials arrive as short codes (DET), abbreviations (DETENT), full descriptions (Detention at Delivery), and vendor-specific SKUs (ACC-0417). There is no industry-standard code set that every carrier honours, so the surface forms are effectively unbounded.
2. No canonical dictionary as the anchor. Without a curated set of canonical codes and their known aliases, every new spelling is a new unknown. Exact match has nothing to be lenient about.
3. Over-eager fuzzy matching is worse than none. The naive fix — accept the closest string match — produces false merges. DETENTION and DENTON (a city) are one edit apart; LIFTGATE and TAILGATE share most of their characters. A blind nearest-neighbour match will confidently fold a distinct charge into the wrong schedule entry, which is harder to catch than an honest UNMAPPED.
4. No confidence threshold or review path. A match is treated as binary when it is really a score. Without a threshold and a queue for the ambiguous middle, you are forced to choose between missing codes and mis-mapping them.

Reproducible Diagnostic

Before adding any fuzzy logic, measure how much your exact-match dictionary is actually missing and what it is missing. This snippet reports the unmapped rate and the raw forms driving it, so you tune against real inputs rather than guesses:

import polars as pl

lines = pl.read_csv("accessorial_lines.csv")   # raw_code, description, carrier_scac
canon = pl.read_csv("alias_dictionary.csv")     # alias, canonical_code

def norm(col: str) -> pl.Expr:
    # uppercase, keep alphanumerics only — "Detention - Loading" -> "DETENTIONLOADING"
    return pl.col(col).str.to_uppercase().str.replace_all(r"[^A-Z0-9]", "").alias("key")

mapped = lines.with_columns(norm("raw_code")).join(
    canon.with_columns(norm("alias")), on="key", how="left"
)
unmapped = mapped.filter(pl.col("canonical_code").is_null())
print("unmapped rate:", round(unmapped.height / mapped.height, 3))
print(unmapped.group_by("raw_code").len().sort("len", descending=True).head(15))

Read the top offenders as a decision table:

If the unmapped pile is dominated by a handful of spelling variants of charges you already have, fuzzy matching will recover most of it — but only with a threshold and a blocklist, or you will trade misses for mis-maps.

Resolution Path

The fix is a layered matcher: normalize, try an exact canonical dictionary first, then fall back to a scored fuzzy match gated by a confidence threshold and a blocklist, and route the ambiguous middle to review. Pin the toolchain:

# requirements.txt
polars==1.12.0
rapidfuzz==3.10.1
pydantic==2.10.6

Step 1 — Normalize and try the exact dictionary first

Fuzzy matching is the expensive, risky path — never reach for it before an exact hit. Normalize aggressively so trivial punctuation and case differences resolve at O(1):

import re

def normalize_code(raw: str) -> str:
    """Uppercase, strip everything but A-Z0-9 so 'Detention - Loading' == 'DETENTIONLOADING'."""
    return re.sub(r"[^A-Z0-9]", "", (raw or "").upper())

class CanonicalDictionary:
    def __init__(self, aliases: dict[str, str]):
        # alias key (normalized) -> canonical code
        self.exact = {normalize_code(k): v for k, v in aliases.items()}

    def exact_lookup(self, raw: str) -> str | None:
        return self.exact.get(normalize_code(raw))

A common mistake is to normalize the input but not the dictionary, so DET-01 in the input never matches DET01 in the alias table. Normalize both sides through the same function.

Step 2 — Build the canonical alias set as the match target

Fuzzy matching needs something curated to match against. Each canonical code owns a list of known aliases; the fuzzy stage scores incoming strings against that list, not against a free-for-all of every string ever seen:

CANONICAL_ALIASES = {
    "DETENTION":   ["DETENTION", "DET", "DETENT", "DETENTIONLOADING", "DTN"],
    "LIFTGATE":    ["LIFTGATE", "LGATE", "LIFT", "LIFTGATEDELIVERY"],
    "RESIDENTIAL": ["RESIDENTIAL", "RESI", "RESDLVY", "HOMEDELIVERY"],
    "REDELIVERY":  ["REDELIVERY", "REDEL", "REDLVY"],
}

def flatten_aliases(mapping: dict[str, list[str]]) -> list[tuple[str, str]]:
    """[(normalized_alias, canonical_code), ...] for scored matching."""
    return [
        (normalize_code(alias), canonical)
        for canonical, aliases in mapping.items()
        for alias in aliases
    ]

Keep aliases per canonical code so a review-queue confirmation appends a new alias and permanently converts a fuzzy hit into a future exact hit — the dictionary learns.

Step 3 — Scored fuzzy fallback with a threshold and a blocklist

Only on an exact miss, score the normalized input against every alias and take the best. Two guards make this safe: a confidence threshold that sends the ambiguous middle to review, and a blocklist that vetoes known confusable pairs outright so a distinct charge is never folded into the wrong schedule:

from rapidfuzz import fuzz, process

# pairs we must NEVER merge even if the string distance is small
BLOCKLIST = {
    frozenset({"LIFTGATE", "TAILGATE"}),
    frozenset({"DETENTION", "DENTON"}),
}

def blocked(raw_norm: str, candidate_canonical: str) -> bool:
    return frozenset({raw_norm, candidate_canonical}) in BLOCKLIST

def fuzzy_map(raw: str, aliases: list[tuple[str, str]],
              accept: int = 92, review_floor: int = 80) -> tuple[str, int, str]:
    """Return (canonical_or_UNMAPPED, score, disposition)."""
    key = normalize_code(raw)
    choices = {alias: canonical for alias, canonical in aliases}
    hit = process.extractOne(key, choices.keys(), scorer=fuzz.token_set_ratio)
    if hit is None:
        return "UNMAPPED", 0, "unmapped"
    alias, score, _ = hit
    canonical = choices[alias]
    if blocked(key, canonical):
        return "UNMAPPED", int(score), "blocked"   # confusable — refuse the merge
    if score >= accept:
        return canonical, int(score), "auto"
    if score >= review_floor:
        return canonical, int(score), "review"      # human confirms, then seeds an alias
    return "UNMAPPED", int(score), "unmapped"

Auto-mapped rows flow straight to the contracted schedule check in Accessorial Charge Taxonomy Mapping; confirmed review rows append an alias so the same variant hits exactly next time. Anything the fuzzy stage does resolve then feeds Accessorial Charge Scoring for dispute weighting.

Verification

Prove the matcher recovers real variants without inventing false merges. These belong in the suite that runs whenever the alias set or thresholds change:

def test_exact_variant_maps_high_confidence():
    aliases = flatten_aliases(CANONICAL_ALIASES)
    code, score, disp = fuzzy_map("DETENT", aliases)
    assert code == "DETENTION" and disp == "auto" and score >= 92

def test_free_text_description_resolves():
    aliases = flatten_aliases(CANONICAL_ALIASES)
    code, _, disp = fuzzy_map("Detention - Loading", aliases)
    assert code == "DETENTION" and disp in {"auto", "review"}

def test_blocklist_prevents_false_merge():
    aliases = flatten_aliases(CANONICAL_ALIASES)
    # LIFTGATE vs TAILGATE are close but must never merge
    code, _, disp = fuzzy_map("TAILGATE", aliases)
    assert code == "UNMAPPED" and disp in {"blocked", "unmapped"}

def test_ambiguous_middle_goes_to_review():
    aliases = flatten_aliases(CANONICAL_ALIASES)
    code, score, disp = fuzzy_map("RESDLV", aliases)
    assert disp in {"review", "auto"}   # near RESIDENTIAL/REDELIVERY, never silently unmapped

In production the health signals are the unmapped rate and the auto-map precision confirmed by the review queue. A rising unmapped rate means a new carrier’s vocabulary is unseen; a review queue that keeps rejecting auto-maps means the accept threshold is too low or the blocklist is short.

Preventive Configuration

Encode the thresholds and confusable pairs so the matcher’s behaviour is reviewable, and make alias growth a product of the review queue rather than a code change:

accessorial_fuzzy_matching:
  accept_score: 92          # token_set_ratio at/above → auto-map
  review_floor: 80          # 80–92 → manual review, seeds a new alias on confirm
  scorer: token_set_ratio   # order-insensitive; handles multi-word descriptions
  blocklist:                # never merge these regardless of string distance
    - [LIFTGATE, TAILGATE]
    - [DETENTION, DENTON]
  learn_from_review: true   # confirmed review maps append an exact alias
  alert_on_unmapped_rate: 0.05   # unmapped share above 5% pages the taxonomy owner

• Exact before fuzzy, always. Every confirmed match becomes an alias, so the exact dictionary grows and the fuzzy path shrinks over time. The lookup table that anchors all of this is documented in Building an Accessorial Charge Lookup Table in Postgres.
• Blocklist is not optional. Any two canonical charges within a few edits of each other belong on the blocklist before fuzzy matching goes live, or the first false merge will be the one you never notice.
• Guard the accept threshold in CI. Assert on a labelled fixture that no blocklisted pair ever auto-maps and that known variants clear the accept score, so a threshold change cannot silently regress into false merges.

FAQ

Why not just take the closest fuzzy match and skip the threshold?

Because the closest match is not always the right one. DETENTION and DENTON differ by a single character, and LIFTGATE and TAILGATE share most of theirs. A blind nearest-neighbour will confidently fold a distinct charge into the wrong schedule, and a false merge validates against the wrong contract entry — harder to catch than an honest UNMAPPED. A threshold plus a blocklist keeps the ambiguous cases in front of a human.

Which RapidFuzz scorer should I use for accessorial codes?

Use token_set_ratio for anything that might be a multi-word description. Short codes like DET compare fine with a plain ratio, but free-text fields such as “Detention at Delivery - Loading Dock” reorder and pad tokens, and token-set scoring is order-insensitive and tolerant of extra words. Normalize to uppercase alphanumerics first so punctuation does not depress the score.

How do I stop hand-mapping the same variants every month?

Make the review queue write back. When an analyst confirms that RESDLVY is RESIDENTIAL, append it as an alias to that canonical code. The next occurrence hits the exact dictionary at O(1) and never reaches the fuzzy stage. Over a few onboarding cycles the exact dictionary absorbs each carrier’s vocabulary and the review volume falls to genuinely novel charges.

What confidence score is safe to auto-map?

With token_set_ratio on normalized codes, an accept threshold around 92 auto-maps clean variants while sending the ambiguous 80-to-92 band to review. Do not lower the accept score to shrink the review queue — that is exactly how false merges start. Shrink the queue by growing the exact alias set from confirmed reviews instead.

Related

• Accessorial Charge Taxonomy Mapping — the parent stage that owns the canonical schedule this matcher resolves codes into.
• Building an Accessorial Charge Lookup Table in Postgres — the persistent store for canonical codes and the aliases the review queue feeds back.
• Accessorial Charge Scoring — weights the now-mapped accessorial for dispute likelihood once it validates against the schedule.

Up one level: Accessorial Charge Taxonomy Mapping · Section: Freight Contract Architecture & Rate Mapping