Memory like a 30B, speed like a 3B

What mixture-of-experts actually changes

A dense transformer runs every parameter for every token. A 30B dense model at 4-bit quantization has to read roughly 17 GB of weights out of memory to produce one token, and memory bandwidth — not compute — is what limits generation on a laptop. That is why dense 30B models feel slow on Apple Silicon regardless of how fast the chip is.

A mixture-of-experts model replaces the feed-forward block in each layer with a set of parallel "experts" and a small router that picks a handful of them per token. Qwen3-30B-A3B, the model I run daily, has about 30.5B total parameters and activates roughly 3B of them for any given token. Different tokens route to different experts, so all 30B must be resident and available — but only ~3B are read per token.

Dense 30B @ Q4: ~17 GB resident, ~17 GB read per token MoE 30B-A3B @ Q4: ~17 GB resident, ~2 GB read per token RAM footprint: set by TOTAL parameters Generation speed: set by ACTIVE parameters

So the headline: you pay 30B prices in RAM and get something much closer to 3B speeds in generation. That is the entire reason a 30B-class model is usable on a laptop at all, and it is why "how many parameters" stopped being a single useful number.

The routing has a cost the arithmetic hides. Which experts a token needs is only known at generation time, and consecutive tokens often need different ones, so memory access is scattered rather than sequential. Real throughput lands well below the bandwidth ceiling the numbers above suggest — expect the ratio to help enormously and not to be the 8x the raw parameter math implies.

Unified memory: the Apple Silicon quirk

On Apple Silicon, CPU and GPU share one pool of memory. There is no separate VRAM to fill and no PCIe transfer to pay for, which is why a 64 GB MacBook can hold a model that would need a serious discrete GPU elsewhere. The catch is that macOS caps how much of that pool the GPU may wire down — around 75% by default — and everything else on your machine competes for the rest.

# see the current GPU wired-memory limit (0 = system default, ~75%) sysctl iogpu.wired_limit_mb # raise it — 56 GB on a 64 GB machine. Resets on reboot. sudo sysctl iogpu.wired_limit_mb=57344

Raise this only if you have measured a model spilling, and leave real headroom. Push it too high and the system starts swapping under memory pressure, which is dramatically worse than a slightly smaller model — you go from tokens per second to seconds per token.

Quantization: what each level costs you

Quantization stores weights at reduced precision. The size arithmetic is straightforward: total parameters times bits-per-weight divided by 8. For a 30.5B-parameter model:

The published Q4_K_M build of Qwen3-30B-A3B lands around 18 GB on disk, close to the arithmetic. Q4_K_M is where almost everyone should be: it fits a 32 GB machine with room for context, and the quality gap to Q8 is small enough that most people cannot pick the outputs apart blind.

The failure mode below Q4 is worth knowing because it is counterintuitive. Aggressive quantization does not primarily erase facts — it erases precision. The model still knows things and gets steadily worse at following formatting instructions, closing brackets, staying in the requested structure, and holding a constraint across a long answer. If a local model suddenly cannot keep to a JSON shape it handled last week, check the quantization level before you rewrite the prompt.

Context length is a second, separate RAM bill

Model weights are the fixed cost. The KV cache is the variable one, and it is the number people forget: every token in the context window stores a key and a value vector in every layer, for the whole session.

bytes/token = 2 × n_layers × n_kv_heads × head_dim × bytes_per_element Qwen3-30B-A3B: 48 layers, 4 KV heads, head_dim 128, fp16 (2 bytes) = 2 × 48 × 4 × 128 × 2 = 98,304 bytes = 96 KiB per token 8,192 tokens × 96 KiB = 0.75 GiB 40,960 tokens × 96 KiB = 3.75 GiB 131,072 tokens × 96 KiB = 12 GiB

Grouped-query attention is doing a lot of work in that formula — only 4 KV heads instead of 32 cuts the cache by 8x versus full multi-head attention. Check your model's own config for these values rather than reusing mine; a model with more KV heads has a dramatically more expensive cache and the difference shows up as an out-of-memory error at exactly the context length you needed.

You can halve the cache by quantizing it. In Ollama that is OLLAMA_KV_CACHE_TYPE=q8_0, which requires flash attention to be enabled. The quality cost of an 8-bit KV cache is very small; q4_0 halves it again and does start to show, particularly on long-context recall.

Ollama settings that actually matter

Most Ollama tuning advice is noise. These five are not:

Set these where they survive a reboot. On macOS, launchctl setenv does not persist, and neither does exporting them in a shell that Ollama was not launched from — the daemon reads its environment at start. Put them in the EnvironmentVariables block of a LaunchAgent plist that starts the server, and then actually verify with ps eww $(pgrep ollama) | tr ' ' '\n' | grep OLLAMA. I lost a week to tuning flags that were never applied because the service was being started by a GUI app that never saw them.

Realistic tokens per second on Apple Silicon

Generation speed on a laptop is bounded by memory bandwidth, so the chip tier matters more than the core count. Roughly: base M-series chips land near 100–120 GB/s, Pro chips in the 150–275 GB/s range, and Max chips at 400 GB/s and up. Divide bandwidth by bytes-read-per-token for a theoretical ceiling, then expect real numbers well below it.

On my M1 Max (400 GB/s) running Qwen3-30B-A3B at Q4_K_M, generation sits in the tens of tokens per second at short context and degrades as the context fills — by 20k tokens in, it is meaningfully slower, because attention cost grows with sequence length even though the weight reads do not. Do not trust anyone's numbers including mine; measure your own, because quantization, context length, and what else is running all move it substantially.

curl -s http://localhost:11434/api/chat -d '{ "model": "qwen3:30b-a3b", "stream": false, "messages": [{"role":"user","content":"Write 300 words about memory bandwidth."}] }' | python3 -c ' import sys, json d = json.load(sys.stdin) print("prefill tok/s:", round(d["prompt_eval_count"] / (d["prompt_eval_duration"]/1e9), 1)) print("gen tok/s:", round(d["eval_count"] / (d["eval_duration"]/1e9), 1))'

The number that actually decides your architecture: prefill

Generation speed is what people benchmark. Prompt processing — prefill — is what determines whether your application feels usable, and it is a completely different regime: compute-bound rather than bandwidth-bound, and on Apple Silicon it runs at a few hundred tokens per second rather than the thousands a datacenter GPU manages.

Work the arithmetic for a retrieval pipeline. A RAG query with 20,000 tokens of retrieved context, prefilling at 400 tokens/sec, spends 50 seconds before the first word of the answer appears. The generation phase afterwards might take four seconds. Your users experience a 54-second response and the tokens-per-second number you optimized was irrelevant to 90% of it.

That single fact should shape the design. Retrieve fewer, better chunks — which is what choosing an embedding model and a reranker properly buys you. Keep the stable part of your prompt stable so the runtime can reuse its cached KV state rather than reprefilling it, the local analogue of prompt caching on a hosted API. And prefer batch work over interactive work wherever the task allows.

When local is genuinely good enough

The honest split is about latency tolerance and correctness stakes, not about capability in the abstract.

My own split is exactly this. Overnight automation drafts on the local model — free retries, no rate limits, and if a draft is bad the cost of regenerating it is zero. Anything with a deadline or a correctness requirement goes to a hosted API, where the current Claude lineup runs from claude-haiku-4-5-20251001 for cheap high-volume work through claude-sonnet-5 and claude-opus-5 for the hard problems. The two are not competitors; they are different tools that happen to share an interface.

A working setup, start to finish

1. Check your RAM. 32 GB runs a 30B MoE at Q4_K_M with modest context. 64 GB runs it comfortably with room for a large window. 16 GB should be running a 7B-class model instead — this guide is not for you and you will be happier.
2. Pull the Q4_K_M build. Start there always; move up only if you can demonstrate a quality problem, and move down only if you cannot fit.
3. Set the five environment variables above in a LaunchAgent plist so they survive a reboot, then verify with ps eww that the running process actually has them.
4. Set num_ctx per request to the smallest value your task needs. Do not configure 128k because it is available.
5. Measure prefill and generation separately with the timing script above, at the context length you actually use.
6. Design around prefill, not generation. Fewer retrieved chunks, stable prompt prefixes, batch where you can.
7. Keep an API path for the work that needs it. A local-only architecture is a constraint, not a virtue.

The setup takes an afternoon and then largely maintains itself. Everything I have built on top of it — the automation drafting, the classification passes, the overnight batch work you can see the shape of on the projects page — runs on that one resident model, and the whole thing costs the electricity to keep a laptop awake.

Tools referenced in this guide

• Local LLM with Ollama — the setup layer beneath this guide — model selection, the HTTP API, and quantisation basics.
• Prompt caching — why keeping the stable part of a prompt stable matters locally and on a hosted API.
• Embedding models for RAG — retrieving fewer, better chunks is the main defence against slow prefill.
• My stack — the actual hardware and tooling behind this.