Running a local LLM, and when it is actually the right call

When local wins

• High-volume, low-difficulty work — drafting, classifying, tagging, extracting fields, first-pass summarising. Thousands of calls a night cost nothing beyond electricity.
• Data that must not leave the machine — anything under an NDA, personal records, or client material.
• Batch jobs that run unattended — no rate limits, no API outage, no bill that scales with a runaway loop.
• Offline environments — flights, bad connections, air-gapped work.
• Anything you want to run a hundred times to test — the cost of iterating on a prompt drops to zero, which changes how much you are willing to experiment.

When local loses: long multi-step reasoning, large-context work, anything where a subtle mistake is expensive, and code generation of any real complexity. Use the frontier API there and stop being precious about it. The failure mode of a local model is not obvious wrongness — it is plausible, well-formatted wrongness, which is much more expensive to catch.

Picking a model for your hardware

The binding constraint is memory, and the arithmetic is not mysterious. Weight size in bytes is roughly parameters × bits-per-weight ÷ 8. A 4-bit quantised 7B model is 7.0 × 10⁹ × 4.83 ÷ 8 ≈ 4.2 GB, and you need room for that plus the KV cache plus the operating system.

The tokens-per-second bands are wide because they depend almost entirely on memory bandwidth, not on the processor. Do not treat them as promises — treat them as the range you should expect before you tune anything.

The speed arithmetic

Token generation is memory-bandwidth-bound, not compute-bound. Every generated token requires reading the active weights out of memory once, so the ceiling is straightforward:

tokens/sec (ceiling) ≈ memory bandwidth ÷ bytes of active weights real throughput ≈ 55% to 70% of that ceiling Worked: 8B model at Q4_K_M = 4.83 GB of weights on 400 GB/s unified memory ceiling = 400 ÷ 4.83 = 82.8 tokens/sec realistic = 82.8 × 0.65 ≈ 54 tokens/sec

Applied across common hardware classes, using the 65% efficiency factor, this is what to expect. Bandwidth figures are the published specifications for each class:

Two conclusions fall out of that table. First, buying more RAM without more bandwidth lets you load a bigger model that runs unusably slowly. Second, the useful comparison between two machines is their bandwidth ratio, not their core count or their benchmark score.

Why mixture-of-experts changes the answer

A mixture-of-experts model holds all its parameters in memory but activates only a fraction per token. The memory cost tracks the total; the speed tracks the active subset. That breaks the usual trade-off in a way that matters enormously on a laptop.

30B total / 3B active, at Q4_K_M: resident in memory = 30e9 × 4.83 ÷ 8 = 18.1 GB read per token = 3e9 × 4.83 ÷ 8 = 1.8 GB On 400 GB/s at 65% efficiency: a dense 30B would run at 400 ÷ 18.1 × 0.65 ≈ 14 tokens/sec the MoE runs at 400 ÷ 1.8 × 0.65 ≈ 144 tokens/sec Same memory footprint. Roughly ten times the throughput.

That is why a 30B-class MoE is the practical recommendation on a 32 GB machine even though a dense 30B technically fits. The trade is that MoE models are generally a little weaker per parameter than a dense model of the same total size — you are buying speed with quality, not getting it free. The hardware specifics are in the MoE on a MacBook guide.

Quantisation, in numbers

Quantisation stores weights at lower numeric precision. The naming looks arcane but decodes simply: the number is the nominal bit width, K denotes the k-quant family, and the _M or _S suffix indicates how much of the model is kept at higher precision. Sizes below are for a 7.0B-parameter model.

The rule that follows from those numbers: a larger model at Q4_K_M beats a smaller model at Q8_0 at the same memory budget. A 14B at Q4 is 8.5 GB against a 7B at Q8 at 7.4 GB, and the 14B wins comfortably. Below 4 bits the curve turns, and the degradation shows up first in exactly the places you will not notice: schema adherence, instruction following, and long-context recall rather than obvious gibberish.

Context length versus KV-cache memory

This is the arithmetic people miss, and it is the usual reason a model that loaded fine yesterday is now swapping. Attention caches a key and a value vector per token per layer, and that cache scales linearly with the context you configured — whether or not you actually use it.

KV bytes per token = 2 × layers × kv_heads × head_dim × bytes_per_element Worked, an 8B model with grouped-query attention: 32 layers, 8 KV heads, head_dim 128, fp16 (2 bytes) = 2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB per token 4k context → 0.50 GiB 8k context → 1.00 GiB 32k context → 4.00 GiB 128k context → 16.00 GiB

Sixteen gigabytes of KV cache on a model whose weights are under five. That is why setting the context window to the maximum the model advertises is a memory decision disguised as a quality decision. Older architectures without grouped-query attention are far worse: a 7B using full multi-head attention with 32 KV heads costs 512 KiB per token, four times as much, so a 32k context alone needs 16 GiB.

Two mitigations are worth knowing. Quantising the KV cache to 8-bit halves it, taking the 32k case from 4.0 GiB to 2.0 GiB for a quality cost that is negligible on most tasks. And flash attention reduces the peak working memory during the forward pass, though not the cache itself. Set the context to what the job needs and no more — a classification prompt does not need 32k, and configuring it anyway costs you gigabytes for nothing.

Use the HTTP API, not the CLI

Ollama exposes a local HTTP endpoint on port 11434. Scripts should call that, not shell out to the interactive command.

curl http://localhost:11434/api/chat -d '{ "model": "your-model", "messages": [ {"role": "system", "content": "You extract fields. Reply with JSON only."}, {"role": "user", "content": "..."} ], "stream": false, "keep_alive": "30m", "options": { "temperature": 0.2, "num_ctx": 8192, "num_predict": 512, "seed": 42 } }'

This matters more than it sounds. The interactive CLI writes progress indicators and terminal control characters to stdout; capturing that from a script produces output that is subtly corrupted in ways that only show up in production, usually as stray escape sequences embedded in a field you are about to write to a database. Automations should always talk to the HTTP API. I learned that the annoying way, and it is now a standing rule in my own tooling.

The options that actually matter

The reasoning-mode item deserves emphasis because it is the most common cause of a local setup that appears to be broken. If the model supports an extended-thinking mode, it will happily spend a long time deliberating on a job that did not need it, and from the outside a request that used to take two seconds simply looks hung. Turn it off for classification, tagging, and extraction.

Keep-alive and cold-start latency

Loading a multi-gigabyte model into memory takes several seconds. If the model is evicted between calls, you pay that on every request, and in a batch job the load time can dominate the actual generation.

Assume an 8-second load and 1,000 tokens generated at 60 tok/s (16.7 seconds of generation per call). 100 calls, model evicted each time: 100 × (16.7 + 8) = 2,470 s = 41.2 minutes 100 calls, model kept resident: 8 + (100 × 16.7) = 1,678 s = 28.0 minutes Overhead removed: 13.2 minutes, about 47% of the generation time.

Set keep_alive per request, or set the server-wide default via the environment. For a machine that runs scheduled batch work, keeping the model resident indefinitely is usually correct — the memory is doing nothing else at 4am. For a laptop you also use for other things, a 5- to 30-minute window is the better trade. If a model is being evicted unexpectedly, check whether something else asked for a different model and forced it out; the loaded-model limit is a separate setting from keep-alive.

Structured output and validation

Local models drift from a schema more than frontier models do, so treat the output as untrusted input from a flaky service. Three layers, in order of how much they buy you:

1. Constrain generation. Ollama accepts a format parameter set either to the string json or to a full JSON schema object. A schema is strictly better, because it constrains decoding rather than merely asking nicely.
2. Validate against the schema anyway. Constrained decoding gets you syntactically valid JSON with the right keys; it does not get you a date that parses, an enum value from your list, or a number in range.
3. Retry with the error attached. Feed the validation failure back as a message and ask for a correction. Cap it at two retries and then fail loudly, because a third attempt almost never succeeds and an infinite retry loop on a free local model is the one way it can still cost you something.

"format": { "type": "object", "properties": { "company": {"type": "string"}, "intent": {"type": "string", "enum": ["quote", "support", "spam"]}, "confidence":{"type": "number"} }, "required": ["company", "intent", "confidence"] } Then, in code: 1. parse -- fails on truncation and stray prose 2. validate -- fails on wrong enum, out-of-range number 3. retry once with the validation error in the prompt 4. on second failure, quarantine the record for a human

Two practical notes. Lower repeat_penalty when generating JSON, because structural characters repeat by definition and penalising them is how you get malformed output. And set a num_predict cap that comfortably exceeds your largest expected object, since a truncated response is indistinguishable from a malformed one at the parser. The general pattern across providers is covered in the structured LLM output guide.

Batching and concurrency

One local model serving many requests behaves differently from an API. Because generation is bandwidth-bound, running several requests in parallel against the same loaded model costs almost nothing extra per additional request — the weights are read once per step regardless of how many sequences are decoding. Up to a point, concurrency is nearly free.

• Parallel requests against one model: good. Set the parallel-request limit to 2–4 and total throughput rises noticeably while per-request latency worsens only slightly.
• Multiple different models loaded at once: usually bad. Each one occupies its own memory, and on a laptop you will hit swap. Keep the loaded-model limit at 1 unless you have obvious headroom.
• KV cache scales with parallelism. Four concurrent slots at 8k context each need four times the KV cache. Re-run the arithmetic above before raising the limit.
• Queue rather than fan out. A simple worker pool of 2–4 against one resident model beats spawning a process per item, which reloads weights and thrashes memory.
• Batch by prompt shape. Grouping similar-length prompts together reduces wasted padding and makes throughput predictable enough to schedule around.

Troubleshooting

That last row is worth a sentence of its own. Environment variables exported in a login shell do not exist for a job started by a system scheduler, and a model server started by hand does not survive a reboot. If a nightly batch silently produces nothing, check that the server is actually running and that its tuning variables are set in the service definition before you debug anything in the prompt.

Cost, as arithmetic

"Local is free" is not quite true and the real number is worth computing, because it settles the argument in both directions. Generating one million tokens at 60 tokens per second takes 4.63 hours. A laptop under sustained inference load draws roughly 60 watts, so:

1,000,000 ÷ 60 tok/s = 16,667 s = 4.63 hours 60 W × 4.63 h = 278 Wh = 0.278 kWh 0.278 kWh × $0.30/kWh = $0.083 per million tokens At 15 tok/s on a bigger model: 18.5 h, 1.11 kWh = $0.33 per million. Against an illustrative hosted rate of $3.00 per million output tokens, local runs roughly 36x cheaper per token on the marginal cost. But on a $3,000 machine bought FOR this: $3,000 ÷ $3.00 per million = 1,000 million tokens to break even at 8 hours a day of continuous 60 tok/s generation = 1.73M tokens/day → about 579 days.

The hosted rate above is illustrative and provider pricing changes; substitute the current figure for whatever you actually use. The structural conclusion holds regardless: if the machine already exists, local is close to free and the decision is purely about capability. If you are buying hardware to save on tokens, the payback period is measured in years and the frontier model will have moved on before you get there.

One honest caveat on the comparison: a local 8B and a frontier model are not interchangeable units, so cost per token is not a like-for-like price. The fair comparison is cost per completed task at acceptable quality, and on hard tasks the local model's cost per completed task is infinite because it never completes them. On the volume tier of work where it does succeed, the arithmetic above is real. For hosted work, prompt caching is usually the larger lever on the bill than model choice.

A realistic architecture

The pattern that works is a two-tier one. A local model does the volume: drafting, tagging, first-pass extraction, overnight batches. A frontier model does the judgement: the final pass, the hard reasoning, anything that ships to a customer. A quality gate sits in between and rejects bad local output before it reaches the expensive step.

1. Local model drafts, tags, or extracts on a schedule, writing to a staging area rather than to anything live.
2. An automated audit runs deterministic checks — schema validity, required phrases present, banned phrases absent, length bounds, duplicate detection.
3. Failures are quarantined with the reason recorded, not silently dropped. The quarantine rate is your quality metric.
4. Survivors go to the expensive tier, or straight to a human, depending on the stakes.
5. Nothing is sent, published, or committed by the pipeline itself. The last step stays manual by design.

The quarantine rate is the number to watch: if it climbs, either the model was swapped, the prompt drifted, or the input distribution changed, and all three are worth knowing about before the output reaches anyone. That is how my own outreach and automation pipelines run. FreightDesk is a production example of the same shape, the small-business AI automation guide covers where this pattern pays off commercially, and RAG is how you give the local half of it access to your own documents. The rest of the stack is here →

Tools referenced in this guide

• My stack — the actual tools behind the trading, the code, and the automation.
• FreightDesk AI — a production example of local-model drafting with a human-grade quality gate.
• RAG explained — how to give a local model access to your own documents.