Picking an embedding model, and why it matters less than you think

The question, and the better question

The question that gets asked is "which embedding model should I use?" It has a boring answer: pick a well-regarded general-purpose model at 768 or 1024 dimensions, ship it, and move on. The question worth asking is "why is my retrieval missing the obviously relevant paragraph?" — and in most broken pipelines the answer has nothing to do with the model. It is a chunk that split a table away from its header, a query phrased as a question against chunks written as statements, or a top-k of 3 when the answer needed 8.

This guide covers the model choice properly, because it does matter at the margins, and then covers the things that matter more. If you are new to the overall pipeline, the RAG explainer is the prerequisite; this one assumes you already know what a vector store is and are now trying to make one work well.

What an embedding actually is

An embedding model is a neural network that reads text and outputs a fixed-length list of floating-point numbers — a vector. A 1024-dimension model always returns 1024 numbers, whether the input was one word or four hundred. The model was trained so that texts humans would consider related land close together in that space, and unrelated texts land far apart. Nothing about the vector is human-readable. Dimension 417 does not mean "legal document." The only thing you can do with a vector is compare it to another vector.

The comparison is almost always cosine similarity: the cosine of the angle between two vectors, which ignores their lengths and measures only direction. It runs from -1 (opposite) through 0 (unrelated) to 1 (identical direction). Most modern embedding models output unit-length vectors, which makes cosine similarity and the dot product the same operation, which is why vector databases can compare millions of vectors per second.

cos(A, B) = (A · B) / (||A|| × ||B||) with unit-length vectors this collapses to just the dot product: cos(A, B) = A · B = a1×b1 + a2×b2 + ... + an×bn

The critical consequence: similar does not mean relevant, and it definitely does not mean correct. An embedding model trained on general web text will happily rank "the refund policy is 30 days" and "the refund policy is 90 days" as near-identical, because they are near-identical as language. Vector search finds text that looks like the query. Everything downstream has to survive that limitation.

The four numbers on a model card that matter

Dimensions: a storage decision, not a quality decision

Dimensions are the number everyone fixates on, and they are mostly a cost lever. Storage is exactly linear and easy to compute: dimensions × 4 bytes for float32, times the number of chunks.

bytes = n_chunks × dimensions × 4 (float32) 10,000 chunks × 768 dims × 4 = 30.7 MB 10,000 chunks × 1536 dims × 4 = 61.4 MB 10,000 chunks × 3072 dims × 4 = 122.9 MB 100,000 chunks × 1024 dims × 4 = 409.6 MB

Then add index overhead. An HNSW graph — the default in most vector databases — typically costs another 1.5× to 2× on top of the raw vectors, and it wants to live in RAM. At 10,000 chunks none of this matters on any machine built this decade. At 5 million chunks, the difference between 768 and 3072 dimensions is the difference between a small instance and a large one, every month, forever.

The quality return on dimensions flattens fast. Going from 384 to 768 is usually a real improvement. From 768 to 1024, small. From 1536 to 3072, often within measurement noise on a domain-specific corpus. Some newer models support Matryoshka representation learning, where you can truncate the vector to a shorter prefix and keep most of the quality — that is a genuinely useful lever if you have it, because it lets you make the storage decision after you have measured, instead of before.

Max sequence length: the silent truncation trap

Every embedding model has a maximum input length, typically 512 tokens for older models and 8,192 for newer ones. Feed it more and most implementations truncate without warning. Your 1,200-token chunk gets embedded as its first 512 tokens, the rest is discarded, and retrieval quietly fails for any query whose answer lived in the back half. There is no error, no log line, and no way to notice except by measuring recall.

This is where model choice and chunking interact, and it is the one interaction you must get right. Chunk to comfortably under the model's limit, and verify by tokenizing a sample of your longest chunks with the model's own tokenizer rather than estimating with a characters-per-token rule of thumb — that rule is wrong by 30% or more on code, tables, and non-English text.

Benchmark scores: a filter, not a ranking

Public retrieval leaderboards are useful for excluding bad models and nearly useless for choosing between good ones. The top twenty models sit within a few points of each other on an average over tasks that are probably nothing like yours, and models get tuned against those benchmarks, which inflates the top of the board relative to real-world behavior. Use the leaderboard to build a shortlist of three. Use your own data to pick from the shortlist.

Local versus API embeddings

The tradeoff is different from the one for generation models, because embedding models are small. A strong general-purpose embedder is often a few hundred million parameters, which runs comfortably on a laptop CPU and very fast on any GPU. You do not need a 30B model to embed a paragraph, which is why local embeddings are much more often the right call than local generation.

Pull and call a local embedder in two commands. Note that this is the HTTP API, not the interactive CLI — for anything programmatic, always talk to the API, which returns structured JSON instead of a stream of terminal formatting you would have to parse:

ollama pull nomic-embed-text curl -s http://localhost:11434/api/embed \ -d '{"model":"nomic-embed-text","input":["first chunk","second chunk"]}' \ | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d['embeddings']), len(d['embeddings'][0]))" # -> 2 768 (two vectors, 768 dimensions each)

One thing that surprises people: Anthropic does not ship an embedding model. Claude is a generation model, and a RAG pipeline built around Claude uses someone else's embedder — local or hosted — for the retrieval half and Claude only for the answer. The two halves are genuinely independent, which is good news: you can change either one without touching the other, as long as you re-index when you change the embedder.

Chunking is the variable you actually control

Here is the honest hierarchy, in descending order of how much each one moves retrieval quality on a typical corpus: how you split the documents, how many results you retrieve, whether you rerank, and only then which embedding model you used. Teams routinely spend a week A/B-testing embedders for a two-point recall difference while running a fixed 512-character split that severs every table from its caption.

What actually works, in rough order of impact:

• Split on structure, not on character count. Headings, sections, list items, function definitions. A chunk should be a thing, not a fixed-width window that happened to land there.
• Keep the chunk self-contained. If it says "this applies to the above," retrieval will surface it alone and the model will have no idea what "above" was. Prepend the document title and section heading to every chunk — it costs a few tokens and it is the single cheapest quality win available.
• Overlap by 10–15%. Enough that a sentence spanning a boundary survives in at least one chunk. More than 20% and you are paying storage to store the same text repeatedly.
• Never split a table, a code block, or a numbered list. These are the chunks most likely to hold the actual answer and the most destroyed by a naive split.
• Aim for 200–500 tokens. Short enough that the embedding is about one thing, long enough to carry context. Well under any modern model's sequence limit.
• Store the parent document ID and offsets. Retrieve the chunk, then optionally expand to its neighbors before handing text to the generation model. Retrieval granularity and context granularity do not have to match.

How to evaluate: recall@k, on fifty of your own questions

Retrieval is measurable, and measuring it takes an afternoon. Build a golden set: fifty real questions someone would actually ask your corpus, each labeled with the chunk or document that contains the answer. Fifty is not statistically luxurious, but it will reliably show you a ten-point difference, which is the size of difference worth acting on.

Then compute recall@k — the fraction of questions where a correct chunk appears anywhere in the top k results. This is the metric that matters, because the generation model does not care whether the right chunk ranked first or fourth, only whether it was in the window at all.

recall@k = (queries where a relevant chunk is in the top k) / (total queries) MRR = mean over queries of 1 / (rank of the first relevant chunk) # recall@k answers: did we retrieve it at all? # MRR answers: how high did it rank? # Optimize recall@k first. Then use a reranker to fix ordering.

A tiny harness is enough. Run the golden set against configuration A, run it against configuration B, print both numbers, and change exactly one thing at a time:

def recall_at_k(store, golden, k=10): hits = 0 for question, expected_id in golden: results = store.search(question, k=k) if any(r.chunk_id == expected_id for r in results): hits += 1 return hits / len(golden) for k in (1, 3, 5, 10, 20): print(k, round(recall_at_k(store, GOLDEN, k), 3))

The shape of that curve tells you what to fix. If recall@20 is high but recall@3 is low, retrieval is finding the right chunk and ranking it badly — that is a reranker problem. If recall@20 is also low, the right chunk is not being found at all, which is a chunking or an embedding problem, and chunking is the far more likely culprit. Diagnosing which of the two you have is the whole point of running the sweep before touching anything.

When a reranker beats a better embedder

An embedding model compares a query vector to a chunk vector — the two were encoded separately and never saw each other. That is what makes vector search fast: you precompute every chunk once. A cross-encoder reranker instead reads the query and the chunk together and scores the pair directly, which is dramatically more accurate and far too slow to run against a whole corpus.

So you use both. Retrieve wide and cheap with vectors, then rerank narrow and expensive with a cross-encoder.

query -> vector search, k=50 (fast, approximate, high recall) -> cross-encoder rerank -> keep top 8 (slow, accurate, high precision) -> generation model 50 cross-encoder scorings per query: tens of milliseconds. 5,000,000 cross-encoder scorings per query: not a product.

In practice a mediocre embedder plus a reranker beats an excellent embedder alone on nearly every corpus, because the reranker is doing a fundamentally harder and more informative computation. If you have a fixed budget of engineering time, adding a reranker is a better use of it than upgrading an embedder. The catch is latency: the rerank step adds real milliseconds to every query, so it belongs in a pipeline where the user is already waiting on a generated answer, not in an autocomplete.

The migration cost nobody budgets for

Vectors from different models are not comparable. Not "slightly less accurate" — meaningless. Different dimensionality, different geometry, no shared reference frame. Changing your embedding model means re-embedding every chunk in the corpus and rebuilding the index, with no partial migration and no fallback.

This is why an API embedding model deprecation is a genuine operational event and not a version bump. Keep the raw source text alongside your vectors, always, so a re-index is a batch job and not an archaeology project. If you only stored the vectors, you cannot migrate — you have to re-crawl the source, and the source may have changed.

A decision table

What I actually run

My own retrieval work runs local end to end: a local embedding model over documents that never leave the machine, and a local generation model via Ollama — currently a 30B mixture-of-experts model at roughly 40k context — for the answering half. The reason is not cost, it is that the corpora involved are personal notes and unpublished research, and "this never touched a third-party API" is a property worth more than two points of benchmark score.

The one place I stopped optimizing the model and started optimizing the pipeline was when I built the retrieval side of an equity research platform during an IBM SkillsBuild program: an NLP scoring system reading 50+ live market sources a day, hitting 78% directional accuracy and cutting manual research time by about 60%. Nearly all of the quality came from how the source documents were segmented and filtered before they were ever embedded. The embedder was the least interesting decision in the system.

Tools referenced in this guide

• RAG explained — the pipeline this guide plugs into — chunking, retrieval, reranking, and where it breaks.
• Local LLM with Ollama — running the generation half on your own machine.
• 30B MoE on a MacBook — what it costs in RAM and tokens/sec to keep the whole pipeline local.
• My stack — the tools behind the retrieval and automation work.