Designing a freight intake schema for messages that were never forms

Freight and logistics intake often begins in text that was written for a person, not a database. A request may come through SMS, WhatsApp, or email; it may contain a route, a pickup time, weight, equipment note, contact, and several unstated assumptions. A parser is useful only when the record it creates helps an operator move forward without hiding uncertainty.

This is the systems-design problem behind IntakeKit, my in-development project for parsing unstructured freight intake messages. IntakeKit is unpublished and has no users; the design goal is to make the first review of a message more organized, not to claim that an ambiguous message can be fully automated.

The schema is the product boundary

Start by defining the smallest record that supports a clear next action. For an initial load inquiry, that might include origin, destination, pickup timing, weight, pallet count, equipment notes, contact details, free-form notes, and the original text. Do not add a field merely because a model can generate something that sounds useful. Every field should have a reason to exist and an owner who can act on it.

{
  "source_text": "",
  "origin": null,
  "destination": null,
  "pickup_window": null,
  "weight_lb": null,
  "pallet_count": null,
  "equipment_notes": [],
  "contact": null,
  "notes": [],
  "field_evidence": {}
}

The critical field is source_text. Keep the incoming message immutable. Normalized values are useful for sorting and searching, but the original words are what an operator needs when a number, date, or place name has been read incorrectly.

Represent missing information honestly

There is a meaningful difference between “not stated,” “candidate found but unverified,” and “confirmed by a person.” A single empty field loses that distinction. Use null for absent values, attach evidence for extracted candidates, and allow a reviewer to mark a value as confirmed or corrected. This makes a record legible when it moves from an intake screen to dispatch or follow-up.

Evidence should travel with the field

A simple evidence object can store the extraction method and the exact phrase that supported a value:

{
  "weight_lb": 1200,
  "field_evidence": {
    "weight_lb": {
      "method": "regex",
      "source_phrase": "1,200 lb",
      "review_status": "unreviewed"
    }
  }
}

This is more helpful than an opaque score. A dispatcher can quickly decide whether a value came from a stable pattern, a local lookup table, a model suggestion, or a human correction. The representation also makes it easier to test changes: a new rule should improve the candidate without overwriting what the message actually said.

Separate deterministic fields from ambiguous ones

Email addresses, phone numbers, and many unit expressions have recognizable shapes. For those, deterministic patterns are usually the first tool: they are fast, explainable, and easy to test against a growing set of real-looking examples. City abbreviations, appointment language, and equipment descriptions carry more context and deserve a different path.

A practical pipeline can run in layers:

1. Preserve the raw message and create a parsing copy.
2. Normalize only stable surface forms, such as repeated spaces or obvious unit spellings.
3. Extract deterministic candidates with patterns.
4. Use a constrained local model only for fields that need contextual reading.
5. Validate types and required fields against the schema.
6. Present missing, conflicting, and low-evidence fields for human review.

The order matters. A model should not be asked to rediscover a phone number that a precise pattern can identify. Conversely, a pattern should not pretend that every three-letter token is a city. Each tool should be used where its failure mode is visible.

Plan for conflicts and revisions

Text threads evolve. An early message may say “tomorrow morning,” followed by a correction; an email reply may contain a new destination. A record therefore needs an event-minded shape: retain prior values, identify the message or phrase that introduced a revision, and let a reviewer choose the active value. Silent overwrites make debugging and customer follow-up harder.

For the same reason, avoid a schema that turns every phrase into a rigid dropdown too early. Preserve the free-form note alongside normalized categories. Operations language is local: an abbreviation that is safe for one team may be misleading for another.

Test the review experience

Evaluation should begin with a human workflow. Can an operator see the original message, identify what is missing, correct a candidate, and understand why it was extracted? A parser that produces neat JSON but sends reviewers back to the message thread for every decision has not earned much trust.

Build a small test corpus of representative message shapes, including incomplete requests, corrections, inconsistent units, and duplicate-looking numbers. Every correction becomes a candidate test case. This turns the system into a maintained operational tool rather than a one-time demo.

For the extraction and validation layer, see Parsing unstructured freight messages into structured load data. For a privacy-oriented architecture, see Local models for private client data.