RAG, without the diagram everyone reposts

What problem does RAG solve?

A language model knows what was in its training data and nothing about your documents, your database, or anything that happened after its cutoff. The two ways to fix that are fine-tuning, which bakes knowledge into weights and is expensive to update, and retrieval, which fetches the relevant text at question time and hands it to the model.

Retrieval wins for facts that change. Fine-tuning wins for behaviour and format. Most people who think they need fine-tuning need retrieval and a better prompt.

The pipeline

1. Chunk — split documents into passages small enough to be precise and large enough to be self-contained.
2. Embed — convert each chunk into a vector that encodes meaning, so similar text lands nearby in vector space.
3. Index — store the vectors so nearest-neighbour lookups are fast.
4. Retrieve — embed the question the same way and pull the closest chunks.
5. Rerank — score the candidates with a stronger model and keep the best few.
6. Generate — put those chunks in the prompt with an instruction to answer only from them, and to say so when they do not contain the answer.

Chunking is where most quality is won or lost

Chunk too small and a passage loses the context that made it meaningful — a paragraph that says "this is not permitted" without the sentence naming what "this" is. Chunk too large and the embedding averages several topics into a vector that is close to nothing in particular.

• Split on structure — headings, sections, list items — before splitting on length.
• Overlap adjacent chunks slightly so a sentence spanning a boundary is not orphaned.
• Attach metadata: source, section title, date. It is needed for both filtering and citation.
• Keep tables and code blocks intact. Splitting them mid-structure destroys their meaning entirely.

Where RAG pipelines actually fail

The second-order lesson: evaluate retrieval separately from generation. If you only measure final answers you cannot tell whether the retriever failed or the model did, and you will tune the wrong half of the system.

When plain vector search is not enough

Vector similarity is good at "find text that means something like this" and bad at relationships — prerequisites, dependencies, ordering, hierarchy. Ask "what do I need to understand before this concept?" and similarity search returns passages that mention the concept, not the ones that precede it.

That is where a graph helps: model the entities and the edges between them, walk the graph to assemble context, and use retrieval to fill in the text. This graph-augmented approach is the subject of one of the two SIGCSE Technical Symposium 2026 papers I co-authored — using knowledge graphs to build adaptive curriculum maps, so a tutoring system can reason about prerequisite structure rather than topical similarity.

The companion paper looked at bilingual coding for inclusive CS learning (DOI 10.1145/3770761.3777339), a mixed-methods IRB study of 60 participants that reported significant pre-to-post gains in learner confidence. More on the research →

A sane starting build

1. Start with 20 real questions and their correct answers. Without an evaluation set you are guessing.
2. Chunk on structure, embed, and retrieve top-20 with hybrid keyword plus vector search.
3. Rerank down to the 3–5 chunks you actually put in the prompt.
4. Require the model to cite the chunk behind each claim, and to refuse when the context is insufficient.
5. Measure retrieval hit rate and answer accuracy separately, then fix whichever is worse.

Tools referenced in this guide

• About & research — the SIGCSE 2026 papers, including the graph-augmented retrieval work.
• Local LLM guide — how to run the generation half of this on your own machine.
• My stack — the tools I actually use for this work.