RAG & Vector Databases
RAG Explained: Retrieval-Augmented Generation from Scratch
Intermediate 25 min For: Beginners / Developers / Researchers
Key takeaways
- ▸RAG retrieves relevant passages from your own documents at question time and puts them in the prompt, so the model answers from evidence instead of memory.
- ▸The pipeline has an offline indexing half (load → chunk → embed → store) and an online query half (retrieve → re-rank → generate with citations).
- ▸Retrieval quality is the bottleneck: chunking, hybrid keyword + vector search and re-ranking usually matter more than the choice of LLM.
- ▸Evaluate retrieval and generation separately: did we fetch the right passages, and is the answer faithful to them?
The problem RAG solves
An LLM's knowledge is frozen in its weights at training time (What Is an LLM?). It does not know your company wiki, yesterday's prices or the contract on your desk — and when asked anyway, it may invent a plausible answer.
Retrieval-augmented generation (RAG), named in Lewis et al. (2020), fixes this with a simple idea: look the information up first, then let the model answer using what was found. It is an open-book exam instead of a closed-book one.
The pipeline at a glance
flowchart LR
subgraph Indexing["Offline: indexing"]
D["Documents"] --> C["Chunk"] --> E["Embed"] --> V[("Vector + keyword index")]
end
subgraph Query["Online: answering"]
Q["User question"] --> QE["Embed query"] --> R["Retrieve top-k (vector + BM25)"]
V --> R
R --> RR["Re-rank"] --> P["Prompt: question + passages + rules"]
P --> L["LLM"] --> A["Answer with citations"]
end1. Load and chunk
Extract clean text from PDFs, web pages, docs and tickets, then split it into chunks of a few hundred tokens, often with some overlap. Chunks that are too large dilute relevance; chunks that are too small lose context. Splitting on headings and paragraphs beats splitting every N characters.
2. Embed and index
An embedding model turns each chunk into a vector so that passages with similar meaning are close together (Tokens and Embeddings). Dense retrievers of this kind (Karpukhin et al., 2020) outperform keyword search on many question-answering tasks. Vectors go into an index that supports fast approximate nearest-neighbour search, commonly HNSW graphs (Malkov & Yashunin, 2016) — in a vector database, Postgres with a vector extension, or simply in memory. Store metadata (source, date, permissions) alongside.
3. Retrieve
At question time, embed the query and fetch the top-k most similar chunks. In practice hybrid search works best: combine vector similarity with classic keyword scoring (BM25), because embeddings are weak at exact names, codes and numbers while keywords miss paraphrases. Apply metadata filters, including access control — never retrieve what the user may not see.
4. Re-rank
A re-ranker (cross-encoder) reads the question and each candidate together and scores relevance much more accurately than vector distance, at higher cost — so it runs only on the top 20–100 candidates. Anthropic reported that adding context to chunks before embedding plus BM25 cut retrieval failures by 49%, and by 67% with re-ranking, on their test sets (Contextual Retrieval).
5. Generate with citations
Build a prompt with the question, the selected passages (with IDs) and explicit rules: answer only from the sources; cite the passage IDs; say "I don't know" if the sources are insufficient. Put the most relevant passages first or last: models use information in the middle of long contexts less reliably (Liu et al., 2023).
RAG vs fine-tuning vs long context
| RAG | Fine-tuning | Long context only | |
|---|---|---|---|
| Fresh / changing facts | ✅ update the index | ❌ retrain | ✅ but re-send every time |
| Private data with permissions | ✅ filter per user | ❌ baked into weights | ⚠️ |
| Citations | ✅ | ❌ | ⚠️ possible |
| Change style / format | ⚠️ | ✅ | ⚠️ |
| Cost per question | low | low | high for large corpora |
Evaluating a RAG system
Measure the two halves separately (Gao et al., 2023 survey common metrics):
- Retrieval: for a set of test questions with known relevant passages, how often is a correct passage in the top-k (recall@k)?
- Generation: is the answer faithful to the retrieved passages, and does it actually answer the question? Frameworks such as RAGAS (Es et al., 2023) automate these checks with LLM judges — validate them against human labels.
Most failures trace back to retrieval, so look at what was retrieved before tuning prompts.
Beyond basic RAG
- Agentic RAG – an agent decides when and what to search, possibly several times (How AI Agents Work); search becomes one of its tools (Tool Calling and MCP).
- GraphRAG – builds a knowledge graph and community summaries to answer "global" questions about a whole corpus (Edge et al., 2024).
Try it without code
Desktop apps such as AnythingLLM and Cherry Studio let you drop in documents and chat with them using local or cloud models — a quick way to see chunking and retrieval in action, and their failure modes. Deeper dives: Vector Databases and Chunking, Hybrid Search and Re-ranking (in progress).
Algorithm cards
| Algorithm | In one line | Original paper |
|---|---|---|
| Retrieval-augmented generation (RAG) | Retrieve relevant passages into the prompt, then generate from that evidence | Lewis et al., 2020 |
| HyDE | Have the model write a hypothetical answer and use its embedding to retrieve real documents | Gao et al., 2022 |
| GraphRAG | Extract an entity graph and community summaries to answer corpus-wide questions | Edge et al., 2024 |
| LLM-as-a-judge | A strong model scores answers against a rubric to evaluate quality and faithfulness | Zheng et al., 2023 |
Must-read papers
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) — where the name RAG and the jointly trained retriever + generator come from. arXiv:2005.11401
- Precise Zero-Shot Dense Retrieval without Relevance Labels (2022) — HyDE: rewrite the query with a generated hypothetical document when no labels exist. arXiv:2212.10496
- From Local to Global: A Graph RAG Approach to Query-Focused Summarization (2024) — GraphRAG: knowledge graphs and community summaries for “what is this corpus about” questions. arXiv:2404.16130
Hands-on project (about 8 hours)
Build RAG Q&A over a set of product docs (for example an open-source project’s official documentation). Prepare 30 test questions with reference answers and source passages; compare three chunking strategies (fixed length, by heading, semantic) and two retrieval methods (pure vector vs keyword + vector hybrid), log retrieval recall, faithfulness and answer relevance with RAGAS, and write a one-page conclusion. See the LlamaIndex docs and Microsoft GraphRAG docs.
Self-check
1. What does RAG “faithfulness” measure?
Whether every claim in the answer is supported by the retrieved passages. It does not measure correctness, only whether the answer strays from the sources.
2. What kind of question is GraphRAG good at?
Thematic, corpus-wide questions (e.g. “which risks do these reports keep raising?”) that chunk-by-chunk retrieval handles poorly.
3. What goes wrong with chunks that are too large or too small?
Too large: unrelated text dilutes retrieval and wastes context. Too small: each chunk lacks context and the answer gets scattered across many chunks.
4. Why does HyDE improve retrieval?
User questions are short and phrased differently from documents; a generated hypothetical answer is closer to real documents in wording and embedding space, so relevant passages are easier to find.
FAQ
RAG or fine-tuning — which should I use?
Use RAG when the model needs facts that change, are private, or must be cited. Use fine-tuning to change behaviour, format or style. They combine well: a fine-tuned model can still use RAG.
Do I need a vector database for RAG?
Not always. Small collections fit in memory or in SQLite/Postgres with a vector extension; dedicated vector databases help when you have millions of chunks, need filtering at scale or many concurrent users. Keyword search (BM25) alone is also a valid retriever.
Doesn't a long context window make RAG unnecessary?
Long contexts help, but stuffing everything in is slow and expensive, and models use information in the middle of long inputs less reliably. Retrieval keeps prompts small, cheap and focused, and gives you citations.
Why does my RAG system still hallucinate?
Usually because retrieval missed the right passage, the chunk lacked context, or the prompt did not instruct the model to answer only from the sources and say when they are insufficient.