Module 23 — Retrieval-Augmented Generation (RAG) Systems

Introduction

A Retrieval-Augmented Generation (RAG) system combines a Large Language Model (LLM) with an external knowledge base to generate accurate, up-to-date, and grounded responses.

Unlike a standalone LLM that relies only on its pretrained knowledge, a RAG system retrieves relevant documents before generating an answer.

This enables applications such as:

• PDF Chatbots
• Enterprise Search
• Customer Support
• Research Assistants
• Legal AI
• Medical Assistants
• Knowledge Management
• Code Documentation Search

A complete RAG pipeline includes:

• Document Processing
• Chunking
• Metadata Extraction
• Embedding Generation
• Vector Indexing
• Retrieval
• Prompt Construction
• Context Injection
• LLM Generation
• Evaluation

In this module, you'll learn:

• Chunking
• Metadata
• Indexing
• Retrieval
• Prompt Construction
• Context Injection
• Evaluation
• Hallucination
• Grounding
• Build a PDF Chatbot

Complete RAG Pipeline

textCopy
1                    PDF / Documents
2                           │
3                    Document Loader
4                           │
5                     Text Extraction
6                           │
7                        Chunking
8                           │
9                    Metadata Creation
10                           │
11                   Embedding Generation
12                           │
13                      Vector Database
14                           │
15                     Similarity Search
16                           │
17                  Retrieved Chunks
18                           │
19                  Prompt Construction
20                           │
21                   Context Injection
22                           │
23                     Large Language Model
24                           │
25                      Final Response

1. Chunking

What is Chunking?

Large documents cannot usually fit into an LLM's context window. Instead, documents are divided into smaller chunks before creating embeddings.

Example

textCopy
1Original Document
2
3↓
4
5Page 1
6
7↓
8
9Paragraph 1
10
11↓
12
13Paragraph 2
14
15↓
16
17Chunk 1
18Chunk 2
19Chunk 3

Common Chunking Strategies

• Fixed-size chunking
• Recursive chunking
• Sentence chunking
• Paragraph chunking
• Semantic chunking

Advantages

• Better retrieval
• Lower embedding cost
• Faster search
• Better context quality

Recursive Character Chunking

pythonCopy
1from langchain_text_splitters import RecursiveCharacterTextSplitter
2
3splitter = RecursiveCharacterTextSplitter(
4    chunk_size=500,
5    chunk_overlap=100
6)
7
8chunks = splitter.split_text(document)
9
10print(len(chunks))

2. Metadata

What is Metadata?

Metadata stores additional information about each document chunk.

Examples

textCopy
1Document:
2Annual_Report.pdf
3
4Metadata
5
6{
7    "page": 15,
8    "section": "Revenue",
9    "author": "Finance Team",
10    "year": 2025
11}

Common Metadata Fields

• File name
• Page number
• Title
• Author
• Date
• Category
• Source URL
• Section

Advantages

• Better filtering
• Source attribution
• Faster retrieval
• Improved explainability

3. Indexing

What is Indexing?

Indexing converts embeddings into a searchable structure.

Pipeline

textCopy
1Documents
2
3↓
4
5Embeddings
6
7↓
8
9Vector Index
10
11↓
12
13Similarity Search

Example using FAISS

pythonCopy
1import faiss
2import numpy as np
3
4dimension = 384
5
6index = faiss.IndexFlatL2(dimension)
7
8index.add(
9    np.array(embeddings).astype("float32")
10)

4. Retrieval

What is Retrieval?

Retrieval finds the most relevant document chunks for a user's query.

Pipeline

textCopy
1Question
2
3↓
4
5Embedding
6
7↓
8
9Vector Search
10
11↓
12
13Top-K Chunks

Example

pythonCopy
1query_embedding = model.encode(
2    ["Explain transformers"]
3)
4
5distances, ids = index.search(
6    np.array(query_embedding).astype("float32"),
7    k=5
8)

5. Prompt Construction

What is Prompt Construction?

The retrieved chunks are combined with the user's question to create a prompt for the LLM.

Template

textCopy
1Context:
2
3{retrieved_documents}
4
5Question:
6
7{user_question}
8
9Answer using only the provided context.

Python Example

pythonCopy
1prompt = f"""
2Context:
3
4{context}
5
6Question:
7
8{question}
9
10Answer only from the context.
11"""

Advantages

• Better answers
• Lower hallucination
• Clear instructions

6. Context Injection

What is Context Injection?

Context injection places the retrieved information directly into the model prompt before generation.

Pipeline

textCopy
1Retrieved Chunks
2
3↓
4
5Prompt
6
7↓
8
9LLM
10
11↓
12
13Answer

Example

textCopy
1Context
2
3Transformer models use self-attention...
4
5Question
6
7What is self-attention?

The LLM now answers using the retrieved knowledge instead of relying solely on pretrained memory.

7. Evaluation

Why Evaluate a RAG System?

A RAG system has two major components:

• Retrieval
• Generation

Both need to be evaluated.

Common Metrics

Evaluation Workflow

textCopy
1Question
2
3↓
4
5Retrieved Chunks
6
7↓
8
9Generated Answer
10
11↓
12
13Ground Truth Comparison

8. Hallucination

What is Hallucination?

Hallucination occurs when an LLM generates information that is incorrect, fabricated, or unsupported by evidence.

Example

textCopy
1Question
2
3Who invented Python?
4
5Bad Answer
6
7Elon Musk invented Python.

Causes

• Missing context
• Weak retrieval
• Ambiguous prompts
• Outdated model knowledge

Reducing Hallucinations

• Better embeddings
• Better chunking
• Hybrid search
• Reranking
• Strong prompt instructions
• Grounding with retrieved evidence

9. Grounding

What is Grounding?

Grounding means ensuring that every generated answer is supported by retrieved documents or trusted sources.

Pipeline

textCopy
1Question
2
3↓
4
5Retrieve Evidence
6
7↓
8
9LLM
10
11↓
12
13Grounded Answer

Example Prompt

textCopy
1Use only the provided context.
2
3If the answer is not available,
4respond with:
5
6"I don't know based on the provided documents."

Advantages

• More reliable answers
• Lower hallucination
• Better trust
• Easier verification

Complete RAG Workflow

textCopy
1User Question
2
3↓
4
5Embedding
6
7↓
8
9Vector Search
10
11↓
12
13Top Documents
14
15↓
16
17Prompt Construction
18
19↓
20
21LLM
22
23↓
24
25Grounded Answer

Practice — Build a PDF Chatbot

Step 1 — Load PDF

pythonCopy
1from langchain_community.document_loaders import PyPDFLoader
2
3loader = PyPDFLoader("transformer_book.pdf")
4
5documents = loader.load()

Step 2 — Split into Chunks

pythonCopy
1from langchain_text_splitters import RecursiveCharacterTextSplitter
2
3splitter = RecursiveCharacterTextSplitter(
4    chunk_size=500,
5    chunk_overlap=100
6)
7
8chunks = splitter.split_documents(documents)

Step 3 — Generate Embeddings

pythonCopy
1from sentence_transformers import SentenceTransformer
2
3embedding_model = SentenceTransformer(
4    "all-MiniLM-L6-v2"
5)
6
7texts = [chunk.page_content for chunk in chunks]
8
9embeddings = embedding_model.encode(texts)

Step 4 — Create FAISS Index

pythonCopy
1import faiss
2import numpy as np
3
4dimension = embeddings.shape[1]
5
6index = faiss.IndexFlatL2(dimension)
7
8index.add(
9    np.array(embeddings).astype("float32")
10)

Step 5 — Retrieve Context

pythonCopy
1question = "What is self-attention?"
2
3query_embedding = embedding_model.encode([question])
4
5distances, ids = index.search(
6    np.array(query_embedding).astype("float32"),
7    k=3
8)
9
10context = "\n".join(
11    texts[i]
12    for i in ids[0]
13)

Step 6 — Generate Final Answer

pythonCopy
1from transformers import pipeline
2
3generator = pipeline(
4    "text-generation",
5    model="gpt2"
6)
7
8prompt = f"""
9Context:
10
11{context}
12
13Question:
14
15{question}
16
17Answer only using the context.
18"""
19
20response = generator(
21    prompt,
22    max_new_tokens=120
23)
24
25print(response[0]["generated_text"])

What You'll Learn

• Load and process PDF documents.
• Split text into retrieval-friendly chunks.
• Generate semantic embeddings.
• Index vectors with FAISS.
• Retrieve relevant document chunks.
• Inject retrieved context into an LLM prompt.
• Build an end-to-end PDF chatbot.

Best Practices for Building RAG Systems

Module Summary

After completing this module, you will be able to:

• Explain the architecture of a complete Retrieval-Augmented Generation (RAG) system.
• Apply document chunking strategies for efficient retrieval.
• Use metadata to improve filtering and source tracking.
• Build searchable vector indexes for semantic retrieval.
• Construct prompts that combine user questions with retrieved context.
• Inject relevant context into an LLM to produce grounded responses.
• Evaluate both retrieval quality and answer quality using appropriate metrics.
• Understand the causes of hallucinations and apply grounding techniques to reduce them.
• Build a complete PDF chatbot using document loading, chunking, embeddings, vector search, and LLM-based answer generation.

Next Module: Module 24 – AI Agents & Tool Calling, where you'll learn function calling, tool use, planning, memory, multi-agent systems, LangGraph, MCP (Model Context Protocol), and build autonomous AI agents capable of interacting with APIs, databases, and external tools.