A local RAG, end to end

You have the pieces: embeddings, chunking, a vector store, and a local model. Here is how they become one working system, all on your own machine.

The pipeline

1. Ingest — load your documents (PDF, Markdown, HTML…), extract the text.
2. Chunk — split by structure, with metadata.
3. Embed — run each chunk through a local embedding model.
4. Index — store the vectors in a vector store.
5. Query — embed the question, retrieve the nearest chunks.
6. Answer — feed the chunks plus the question to the model, and ask it to answer from the provided text.

The stack, concretely

• Extraction — a library per format (PDF, docx, markdown).
• Embedding model — a small local model, run through llama.cpp or a dedicated server.
• Vector store — from an in-memory index to something like a local Qdrant/Chroma/LanceDB. For personal use, start with the simplest that stores to disk.
• Model — the LLM that writes the final answer.

What makes it work

1. Ground the answer. Tell the model to answer only from the retrieved chunks, and to say when it does not know. This is what keeps it from inventing.
2. Retrieve enough, not too much. Feed the top few relevant chunks, not fifty. A huge context of vaguely relevant text is worse than a tight one.
3. Cite. Keep the source of each chunk so the answer can say where it came from.

The honest expectation

A local RAG is not a magic oracle. It is a search engine with a good writer on top. It works best when your questions have answers in the documents, and it fails when the answer is not there, so the model either admits it or hallucinates. The next lesson is about those failure modes.

Build the pipeline small and real first: one folder of documents, one class of question, and a way to see what was retrieved. Then grow it.