Running a local LLM, and when it is actually the right call

When local wins

• High-volume, low-difficulty work — drafting, classifying, tagging, extracting fields, first-pass summarising. Thousands of calls a night cost nothing.
• Data that must not leave the machine — anything under an NDA, personal records, or client material.
• Batch jobs that run unattended — no rate limits, no API outage, no bill that scales with a runaway loop.
• Offline environments — flights, bad connections, air-gapped work.

When local loses: long multi-step reasoning, large-context work, anything where a subtle mistake is expensive, and code generation of any real complexity. Use the frontier API there and stop being precious about it.

Picking a model for your hardware

The binding constraint is memory. A rough guide for quantised weights:

Mixture-of-experts models are worth knowing about: they hold a large parameter count in memory but only activate a fraction per token, so throughput feels much closer to a small model than the parameter count suggests. That is the sweet spot on a well-specced laptop.

Quantisation, briefly

Quantisation stores weights at lower precision — 4-bit and 5-bit variants being the common choices — which shrinks memory use several-fold for a modest quality loss. Practical rule: a larger model at 4-bit usually beats a smaller model at 8-bit at the same memory budget. Below 4-bit, quality degrades quickly.

Use the HTTP API, not the CLI

Ollama exposes a local HTTP endpoint on port 11434. Scripts should call that, not shell out to the interactive command.

curl http://localhost:11434/api/generate -d '{ "model": "your-model", "prompt": "Summarise this in one sentence: ...", "stream": false, "options": {"temperature": 0.2} }'

This matters more than it sounds. The interactive CLI writes progress and control characters to stdout; capturing that from a script produces output that is subtly corrupted in ways that only show up in production. Automations should always talk to the HTTP API. I learned that the annoying way, and it is now a standing rule in my own tooling.

Settings that actually matter

• temperature — 0.1–0.3 for extraction and classification, higher only when you want variety.
• context length — bigger contexts cost memory and speed. Set it to what the job needs, not to the maximum.
• reasoning / thinking modes — if the model supports an extended-thinking mode, it will happily spend a long time on a job that did not need it. Disable it for high-volume mechanical work; a request that used to take seconds can otherwise appear to hang.
• keep-alive — reloading weights per call dominates latency in batch jobs. Keep the model resident.
• structured output — ask for JSON and validate it. Local models drift from a schema more than frontier models do, so parse defensively and retry.

A realistic architecture

The pattern that works is a two-tier one. A local model does the volume: drafting, tagging, first-pass extraction, overnight batches. A frontier model does the judgement: the final pass, the hard reasoning, anything that ships to a customer. A quality gate sits in between and rejects bad local output before it reaches the expensive step.

That is how my own outreach and automation pipelines run — local model drafts overnight, an automated quality audit quarantines anything that fails its checks, and only the passing output moves forward. The rest of the stack is here →

Tools referenced in this guide

• My stack — the actual tools behind the trading, the code, and the automation.
• FreightDesk AI — a production example of local-model drafting with a human-grade quality gate.
• RAG explained — how to give a local model access to your own documents.