RAG, without the diagram everyone reposts

What problem does RAG solve?

A language model knows what was in its training data and nothing about your documents, your database, or anything that happened after its cutoff. The two ways to fix that are fine-tuning, which bakes knowledge into weights and is expensive to update, and retrieval, which fetches the relevant text at question time and hands it to the model.

Retrieval wins for facts that change. Fine-tuning wins for behaviour and format. Most people who think they need fine-tuning need retrieval and a better prompt.

The pipeline

1. Chunk — split documents into passages small enough to be precise and large enough to be self-contained.
2. Embed — convert each chunk into a vector that encodes meaning, so similar text lands nearby in vector space.
3. Index — store the vectors so nearest-neighbour lookups are fast.
4. Retrieve — embed the question the same way and pull the closest chunks.
5. Rerank — score the candidates with a stronger model and keep the best few.
6. Generate — put those chunks in the prompt with an instruction to answer only from them, and to say so when they do not contain the answer.

Chunking is where most quality is won or lost

Chunk too small and a passage loses the context that made it meaningful — a paragraph that says "this is not permitted" without the sentence naming what "this" is. Chunk too large and the embedding averages several topics into a vector that is close to nothing in particular.

• Split on structure — headings, sections, list items — before splitting on length.
• Overlap adjacent chunks slightly so a sentence spanning a boundary is not orphaned. 10–20% of the chunk size is the usual range.
• Attach metadata: source, section title, date, and any field you will want to filter on. It is needed for both filtering and citation, and adding it later means re-indexing everything.
• Keep tables and code blocks intact. Splitting them mid-structure destroys their meaning entirely, and a half-table is worse than no table because it looks answerable.
• Prepend the document title and section heading to the chunk text before embedding. It costs about 40 tokens and it fixes an enormous number of "right document, wrong section" misses.
• Chunk sizes in tokens, not characters. English runs roughly four characters per token, so a 512-token chunk is about 2,000 characters or 300–400 words.

For scale, a 500-document corpus of eight pages at 500 words a page is 2,000,000 words. At 400 words per chunk that is 5,000 chunks, and at 1,024 dimensions in float32 the whole index is 20.5 MB. Index size is almost never the constraint at the scale most people are building for — chunk quality is.

Choosing an embedding model

The temptation is to grab the highest-scoring model on a public leaderboard. Six criteria matter more, roughly in this order:

1. Does it match your domain? A general model on legal, medical, or freight-industry text will underperform a mediocre model that saw that vocabulary. Test on your own 20 questions before you read any leaderboard.
2. Sequence limit versus your chunk size. A model with a 512-token limit silently truncates a 768-token chunk, and the part it drops is the end — which is where conclusions live.
3. Symmetric or asymmetric. Question-to-passage retrieval is asymmetric. Models trained for it often need a prefix on the query, on the document, or on both. Getting this wrong quietly costs a large fraction of your accuracy and produces no error.
4. Dimensionality, which is a storage and speed decision. See the table below; 768 or 1,024 is the usual sweet spot, and Matryoshka-style models let you truncate dimensions with graceful degradation.
5. Local or hosted. A hosted model means every document and every query leaves your machine, and it means re-indexing if the provider deprecates the model. A local model means the index is reproducible forever.
6. Stability. Changing the embedding model invalidates the entire index. Treat the choice as a migration, and store the model name and version alongside every vector so you can tell what is stale.

Quantising vectors to int8 cuts storage by four with a small recall cost, and binary quantisation cuts it by thirty-two and is usually paired with a full-precision rescoring pass over the top few hundred candidates. Neither matters below a million chunks. The selection criteria in more depth are in the embedding model guide.

Hybrid search: BM25 plus vectors

Vector search fails on exactly the queries that look easiest: part numbers, error codes, proper nouns, rare acronyms, anything where the literal token is the thing being asked about. Embeddings are trained to collapse surface form into meaning, which is the wrong behaviour when the surface form is the answer. Keyword search handles those perfectly and fails on paraphrase. Running both and fusing the results costs very little and removes the worst failure of each.

The fusion method that works without tuning is Reciprocal Rank Fusion. It ignores the scores — which are not comparable across a BM25 index and a vector index — and uses only the ranks:

RRF(d) = Σ over each ranked list L of 1 ÷ (k + rank of d in L) k = 60 is the standard constant. It damps the top of each list so a single first-place hit cannot dominate the fusion.

Worked, with a keyword list ranking A, B, C, D, E and a vector list ranking C, A, F, B, G:

Notice what the fusion did. A appeared near the top of both lists and wins even though it was first in only one. C was only third on keywords but first on vectors and finishes second. F, which keyword search never saw at all, still beats D, which vector search never saw — because a rank-3 appearance in one list outweighs a rank-4 appearance in the other. Agreement across two different retrieval mechanisms is a stronger signal than a high score in either one.

The alternative is a weighted linear combination of normalised scores, which can perform slightly better but requires tuning the weight on a validation set, and the tuned weight rarely transfers to a new corpus. Start with RRF; move to weighted fusion only if your evaluation set says it helps.

Reranking: what it costs and what it buys

Retrieval embeddings are bi-encoders: the query and the document are encoded separately, so document vectors can be precomputed and a search over a million of them is one vector operation. A reranker is a cross-encoder: it reads the query and one candidate together, which is far more accurate and cannot be precomputed. That is the entire trade — accuracy for a forward pass per candidate.

Latency figures above assume roughly 15 ms per query-document pair on a modest CPU and are illustrative, not benchmarks — a small GPU changes them by an order of magnitude. The budgeting logic is what transfers: if your end-to-end target is 2 seconds and generation takes 1.2, you have 800 ms for retrieval plus reranking, and reranking 50 candidates at 750 ms only just fits.

Reranking is the highest-leverage single addition to a naive pipeline, because it fixes the specific failure where the right chunk was retrieved at rank 14 and your prompt only had room for five. It does nothing at all for a chunk that was never retrieved — a reranker cannot rescue what the retriever missed, which is why hybrid search comes first in the build order.

Context-window budgeting

"Put the chunks in the prompt" hides an allocation problem. The window has to hold the system prompt, the question, the retrieved chunks, and the answer, and if you do not reserve space for the answer the model will be truncated mid-sentence.

Window 8,192 tokens − system prompt 250 − user question 60 − answer allowance 800 − safety margin 150 ──────────────────────────────── available for chunks 6,932 tokens At 512-token chunks + 40 tokens of metadata header = 552 each: 6,932 ÷ 552 = 12 chunks fit At 1,024-token chunks + 40 = 1,064 each: 6,932 ÷ 1,064 = 6 chunks fit At 256-token chunks + 40 = 296 each: 6,932 ÷ 296 = 23 chunks fit

Fitting twelve chunks does not mean sending twelve. Retrieval quality falls off fast down the ranked list, so chunks 6 through 12 are usually diluting the prompt with near-misses rather than adding evidence. Retrieve broadly, rerank hard, and send the best three to five. The remaining budget is better spent on a longer answer allowance and on the metadata headers that make citation possible.

On a local model the arithmetic bites harder, because the configured context window sets KV-cache memory whether you use it or not — the memory maths for that is in the local LLM guide. If the same system prompt and the same instructions lead every request, prompt caching is the cheapest optimisation available on the hosted side.

Where RAG pipelines actually fail

A diagnosis order that saves time

1. Is the answer in the corpus at all? Grep for it. If it is not there, no amount of pipeline work helps, and this is the single most common cause of a "RAG is broken" report.
2. Does the correct chunk appear anywhere in the top 50? If not, it is a retrieval problem. Go to step 3. If it does, it is a ranking or generation problem. Skip to step 5.
3. Does keyword search alone find it? If yes, your vector search is the weak half — add BM25 and fuse. If no, the chunk boundaries or the embedding model are wrong.
4. Is the correct text split across two chunks? If so, it is a chunking problem, not a retrieval one. Increase overlap or move to structure-aware splitting.
5. Is the correct chunk in the top 50 but not the top 5? That is precisely what reranking fixes.
6. Is the correct chunk in the prompt and the answer still wrong? Now it is a generation problem. Tighten the instruction to answer only from context, require a citation per claim, and check whether the context is so long that the middle of it is being ignored.
7. Did the model invent an answer from an empty retrieval? Add and test an explicit refusal path. This must be a test case, not a hope.

The second-order lesson: evaluate retrieval separately from generation. If you only measure final answers you cannot tell whether the retriever failed or the model did, and you will tune the wrong half of the system for a week.

An evaluation harness you can build in an afternoon

Twenty real questions, each with the correct answer and the identifier of the chunk that contains it. That is the whole harness, and it is the difference between engineering and guessing. Three metrics, measured separately:

• Hit rate at k — the fraction of questions where the correct chunk appears in the top k retrieved. Measure at k = 1, 3, and 5. This is a pure retrieval metric and it does not involve the model at all.
• Mean Reciprocal Rank — the average of 1 divided by the rank of the first correct chunk, scoring zero when it does not appear. It rewards putting the right chunk first rather than merely somewhere.
• Answer accuracy — graded against the reference answer, either by hand or by a stronger model with a strict rubric. Run it twice: once with retrieved context, once with the correct chunk injected by hand. The gap between the two is exactly your retrieval deficit.

Five questions. Rank of the first correct chunk: Q1: 1 Q2: 3 Q3: 1 Q4: not found Q5: 2 Reciprocal ranks: 1.000, 0.333, 1.000, 0.000, 0.500 MRR = (1.000 + 0.333 + 1.000 + 0.000 + 0.500) ÷ 5 = 2.833 ÷ 5 = 0.567 hit rate @1 = 2 of 5 = 40% hit rate @3 = 4 of 5 = 80% hit rate @5 = 4 of 5 = 80%

Read those three numbers together. Hit rate at 5 equals hit rate at 3, so retrieving more chunks past three is buying nothing — the miss on Q4 is a genuine retrieval failure that a bigger k will not fix. Meanwhile hit rate at 1 is only 40% against 80% at 3, which is the exact signature of a ranking problem, and ranking problems are what rerankers solve. The metric pattern tells you which fix to reach for.

Two habits make the harness durable. Freeze the question set and version it, so a change is comparable against last week rather than against a moving target. And add every production failure to it as a new case, which turns bug reports into regression tests and stops you fixing the same class of miss twice.

When plain vector search is not enough

Vector similarity is good at "find text that means something like this" and bad at relationships — prerequisites, dependencies, ordering, hierarchy. Ask "what do I need to understand before this concept?" and similarity search returns passages that mention the concept, not the ones that precede it. The relationship you need is not encoded in the text of either passage; it exists between them.

That is where a graph helps: model the entities and the edges between them, walk the graph to assemble context, and use retrieval to fill in the text. The two are complements rather than alternatives — the graph decides which nodes belong in the context and in what order, and vector retrieval supplies the prose for each node.

Education is the clearest example, and it is what I built Prometheus Tutor around: a tutor that models prerequisite concepts as a graph, so it routes a learner to the upstream idea their wrong answer actually implicates rather than re-explaining the question they missed. Asked what a student should study next, similarity search returns the most topically related material, while a prerequisite graph returns the material that actually unlocks it. Those are different answers, and only one of them is useful.

The peer-reviewed SIGCSE Technical Symposium 2026 poster abstract looked at bilingual coding for inclusive computer science learning (DOI 10.1145/3770761.3777339), a mixed-methods IRB study of 60 participants. It found statistically significant pre-to-post gains in programming confidence, computing identity, enjoyment and motivation across those 60 participants, with novices gaining significantly more than experienced programmers. The bilingual-versus-English comparison specifically showed effect sizes in the Cohen's d 0.25 to 0.40 range and was not statistically significant. More on the research →

The cost side is worth stating plainly, because graph RAG is over-recommended. Building the graph means entity extraction, relationship extraction, and de-duplication across the corpus, which is a large one-time job and an ongoing maintenance burden every time documents change. Reach for a graph when the questions are genuinely relational and you can name three of them. If your questions are lookups, hybrid search plus a reranker will beat a badly maintained graph every time.

A sane starting build

1. Write 20 real questions with correct answers and the chunk that contains each. Do this first. Without an evaluation set you are guessing, and every step below becomes unfalsifiable.
2. Chunk on structure, capped at 512 tokens, with 10% overlap, and prepend the document title and section heading to each chunk before embedding.
3. Embed with a model you have tested on your own questions, not on a leaderboard. Store the model name and version with every vector.
4. Retrieve top-20 with hybrid search — BM25 and vectors, fused with RRF at k = 60. Measure hit rate at 20 now; that is your ceiling and nothing downstream can exceed it.
5. Rerank down to the 3 to 5 chunks you actually put in the prompt. Measure hit rate at 5 before and after so you know what the reranker bought.
6. Require a citation per claim, and an explicit refusal when the context is insufficient. Test the refusal path deliberately with a question you know is not in the corpus.
7. Measure retrieval and generation separately, then fix whichever is worse. Re-run the whole harness after every change, and add each production failure to the question set.

Resist adding query rewriting, multi-hop retrieval, agentic loops, or a graph until the harness says the simple pipeline is the bottleneck. Every one of those adds latency and a new failure mode, and most of them are attempts to compensate for chunking that was wrong in step two. Running the generation half locally is a reasonable way to iterate for free while you tune the retrieval half, and where this pattern actually pays off commercially is a separate question worth answering before you build anything.

Tools referenced in this guide

• About & research — the SIGCSE 2026 paper, plus the graph-augmented retrieval project.
• Local LLM guide — how to run the generation half of this on your own machine.
• My stack — the tools I actually use for this work.