Parsing Freight Messages into Load Data

Freight messages are not forms. They arrive as a mix of shorthand, context, phone calls transcribed into text, forwarded email chains, and WhatsApp messages typed while someone is moving between tasks. One message might say, “Need 2 pallets SJ to Reno tomorrow am, 1,200 lb, call Mike,” while another hides the same information across three replies. A useful parsing system does not pretend that this language is clean. It gives the operations team a fast, reviewable starting point.

Start with a schema, not a model

The common failure mode is to ask a language model to “extract the load details” before deciding what a load record actually is. Begin with a small schema that separates required, optional, and derived fields. For an initial intake, that might mean origin, destination, pickup date or window, commodity or equipment notes, weight, pallet count, contact, and the raw message.

{
  "origin": null,
  "destination": null,
  "pickup_window": null,
  "weight_lb": null,
  "pallet_count": null,
  "contact": null,
  "notes": [],
  "source_text": ""
}

The schema is a contract. It prevents the parser from inventing a field because it sounds plausible, and it lets downstream systems distinguish an absent value from an unverified one. Keep the original message permanently attached to the record. A clean set of columns is convenient; the source text is what makes correction possible.

Normalize conservatively

Normalization makes patterns easier to recognize, but over-normalization can erase meaning. Preserve the original string, then create a parsing copy. Convert repeated whitespace to a single space, standardize obvious unit spellings, and expand only abbreviations your team truly uses. “SJ” may mean San José in one operation and something entirely different in another. A lookup table should be local to the business and reviewable by the people who use it.

import re

def normalize(text: str) -> str:
    text = text.strip()
    text = re.sub(r"\s+", " ", text)
    text = re.sub(r"(?i)\blbs?\b", "lb", text)
    text = re.sub(r"(?i)\bpal\b", "pallet", text)
    return text

Notice what this function does not do: it does not convert every city nickname, guess dates, or rearrange clauses. Those transformations may be useful later, but they should remain visible and reversible.

Use deterministic extraction for deterministic fields

Phone numbers, email addresses, weight expressions, and many reference IDs have stable shapes. Regex is usually faster, cheaper, and easier to test than an LLM for these cases. A pattern does not need to solve every message; it needs to pull out high-confidence candidates and report what it found.

WEIGHT = re.compile(r"\b(?P<value>[\d,]+)\s*(?P<unit>lb|lbs|pounds?)\b", re.I)
PALLETS = re.compile(r"\b(?P<count>\d+)\s*(?:pallets?|skids?)\b", re.I)

def parse_weight(text):
    match = WEIGHT.search(text)
    if not match:
        return None
    return int(match.group('value').replace(',', ''))

Test these patterns against awkward input: “1200lbs,” “1.2k lb,” “2 pallet?” and messages where the number belongs to a street address or reference number. The right response to an uncertain match is not to quietly coerce it. Capture the candidate and label it for review.

Let a model handle ambiguity, inside guardrails

Origins, destinations, appointment windows, and operational notes are often ambiguous. A small language model can help identify likely values, but it should receive a narrow task: return only the agreed JSON keys, quote the source phrase when possible, and use null rather than guessing. Validate the output against the schema before it reaches a database.

instruction = """Extract load fields from this message.
Return JSON only. Use null if the text does not support a field.
Do not infer a city, date, or unit that is not stated."""
# model_output = local_model.generate(instruction + message)
# record = validate_against_schema(model_output)

Model output is a candidate, not a fact. If the model returns “San Jose” from “SJ,” attach the exact source span or a warning that an abbreviation map was applied. That small bit of provenance is valuable when a dispatcher asks, “Where did this value come from?”

Build confidence around the record, not a magic score

A single confidence number can hide the important distinction between a phone number matched by a precise pattern and a destination inferred from context. Store confidence or method per field: regex, lookup, model, or human. A UI can then prioritize the fields that deserve attention instead of making a user inspect every value.

Require a human review when a required field is missing.
Flag conflicting values, such as two different pickup dates.
Never overwrite the source message with normalized text.
Log corrections; they become the best future test cases.

Measure the handoff, not just extraction accuracy

The metric that matters is whether a person can move from an incoming message to a reliable next action with less friction. Track fields that are repeatedly corrected, messages that cannot be parsed, and time spent waiting on clarifying questions. This exposes where the schema, lookup table, or workflow needs attention.

For privacy-sensitive operations, the same architecture can run locally: deterministic patterns first, a local small model for ambiguous language, and an approval step before any record is used. That approach is the design direction behind IntakeKit, which is currently in development. It is less glamorous than a fully autonomous agent, but it is more likely to earn trust from the people who have to make the shipment happen.