Run small language models locally so client data never leaves the building

Every document-automation project starts with an attractive promise: take a messy message, ask a model what it means, and receive tidy structured data. The difficult question comes next: where does the text go? For organizations handling help requests, service notes, emails, or intake messages, sending every raw document to a hosted service may be unnecessary. A small local language model can cover a useful subset of extraction work while keeping the data path inside the organization.

Local-first is an architecture, not a slogan

Local-first means the normal extraction loop runs on a machine the organization controls. The message enters a local queue, a parser produces structured candidates, and a reviewer approves or corrects them. The raw document, intermediate prompts, and output remain in the same environment unless an explicit policy says otherwise.

This is different from claiming that local is automatically secure. A local system still needs access control, encrypted storage where appropriate, backups, logging limits, and a plan for deleting data. The advantage is control: the organization can make those choices without making an external model endpoint part of the default workflow.

Choose tasks a small model can actually do well

Small models are most useful when the job is constrained. Extracting a few named fields, classifying a request into a controlled set of categories, or summarizing a short message for an internal reviewer is different from asking for open-ended judgment. Make the prompt short, specify output types, and ask the model to return null when support is missing.

schema = {
  "requester_name": "string | null",
  "phone": "string | null",
  "requested_service": "string | null",
  "preferred_time": "string | null",
  "evidence": "object"
}

prompt = """Extract only the fields in this schema.
Return valid JSON. Do not guess. For every non-null value,
include the exact phrase from the message in evidence."""

The evidence field changes the conversation. Instead of treating the model as an oracle, the system asks it to point back to the words that justify each value. A reviewer can then verify a field in seconds rather than re-reading the entire message.

Use an extraction ladder

A practical local pipeline does not give every task to the model. Start with the least ambiguous method and escalate only when needed:

1. Text cleanup: preserve the original; normalize whitespace and obvious encoding issues in a working copy.
2. Rules: use regex or dictionaries for emails, phone numbers, IDs, and fixed vocabulary.
3. Local model: request constrained JSON for fields that require sentence-level interpretation.
4. Validation: verify JSON shape, allowed values, date formats, and cross-field consistency.
5. Human review: send missing, conflicting, or low-evidence items to a person.

This approach is often faster than a model-only system. It also creates clear test surfaces. You can test a regex for phone extraction independently, then test model prompts against a stable set of difficult messages.

Validate like the model will occasionally be wrong

Because it will. A well-designed system assumes that a fluent output can still be unsupported or malformed. Parse the output with a strict schema validator and reject extra fields. Check that values fit expected patterns, but do not equate a syntactically valid value with a true one. “03/04” is a valid date fragment; it may not be enough information to schedule anything.

def require_supported(record):
    for field, value in record.items():
        if field == "evidence" or value is None:
            continue
        if not record["evidence"].get(field):
            raise ValueError(f"Missing evidence for {field}")
    return record

Validation rules should preserve uncertainty. If a message says “next Tuesday,” keep that phrase as written or mark it for resolution rather than converting it to a calendar date based on an invisible system clock.

Keep model operations observable

Local does not mean opaque. Log the model version, prompt version, parser method, validation result, and reviewer correction without retaining more sensitive content than necessary. When a field is wrong, you need to know whether the error came from normalization, a rule, the model, or a lookup table. Versioning lets you compare changes instead of guessing why accuracy shifted.

Tests should look like the real inbox

Build a test set from de-identified examples or carefully composed cases that represent the formats your staff actually sees: truncated messages, two languages in one note, contradictory times, unconventional phone formatting, and messages with no actionable request. Include expected nulls. A parser that returns nothing when evidence is absent can be more useful than one that confidently fills every field.

Decide when not to automate

Some messages require policy judgment, a sensitive conversation, or context that does not belong in a pattern matcher. A local model can prepare a draft, but it should not make an irreversible decision simply because it can produce a plausible sentence. Put that boundary in the workflow, not in a vague instruction.

IntakeKit is Yusuf Gadelrab’s in-development project exploring this pattern for WhatsApp, SMS, and email requests. It uses a small local model with a regex fallback and has ten passing tests; it is not published and has no users. For the message-structure side of the problem, see the freight intake parsing guide.