Integrating Knowledge Bases for RAGs

GitHub Repository: Brianletort/KBs-For-RAGs

Table of Contents

1. Overview and Technology Stack
2. How Knowledge Bases Power RAG
   • Foundational Architecture
   • Key Advantages
   • Common Challenges
3. Demo 1 — Basic RAG Pipeline (Clip 1)
   • Project Structure
   • Docker Configuration — Qdrant
   • RAG Engine — rag_engine.py
   • FastAPI — main.py
4. Building and Integrating Knowledge Bases
   • Metadata and Labeling
   • Retrieval Optimization
5. Demo 2 — Baseline vs Optimized Comparison (Clip 2)
   • Compared Parameters
   • Improved RAG Engine — rag_engine.py (Clip 2)
6. Optimization and Maintenance at Scale
   • Monitoring and Maintenance
   • Key Production Metrics
7. Demo 3 — Production Features (Clip 3)
   • The 5 Production Improvements
   • Hybrid Retrieval — BM25 + Vector Search
   • Reranking with Cross-encoder
   • Optimized RAG Engine — rag_engine_optimized.py
   • Extended FastAPI — main.py (Clip 3)
8. Three-Clip Comparison Table
9. Installation and Quick Start
10. Key Demonstrated Concepts
11. Additional Resources

1. Overview and Technology Stack

This course teaches how to transform enterprise documentation into queryable, AI-powered knowledge bases using LlamaIndex and Qdrant.

Learning Objectives

• Understand how knowledge bases power RAG systems
• Build and integrate knowledge bases: document preprocessing, embedding generation, and storage in vector databases
• Optimize and maintain RAG systems: indexing, metadata filtering, hybrid search, and reranking

Technology Stack

Repository Structure

KBs-For-RAGs/
├── Clip1-v2/          # Basic RAG pipeline
│   ├── backend/       # FastAPI + LlamaIndex engine
│   ├── frontend/      # Next.js chat interface
│   ├── data/          # IT knowledge base CSV files
│   └── docker/        # docker-compose for Qdrant
├── Clip2/             # Optimization and comparison
│   ├── backend/       # Improved RAG (chunking, embeddings)
│   ├── frontend/      # Baseline vs optimized comparison UI
│   └── data/          # Extended knowledge base
└── Clip3/             # Production features
    ├── backend/       # Hybrid search + reranking
    ├── frontend/      # Enhanced UI with metrics
    └── data/          # Knowledge base with rich metadata

2. How Knowledge Bases Power RAG

2.1 Foundational Architecture

A knowledge base in a RAG system combines four essential elements:

graph TD
    A[Structured Data\ncategories, systems,\npriorities, tags] --> D[Unified\nKnowledge Base]
    B[Unstructured Data\ndocuments, guides,\npolicies, long text] --> D
    C[Consolidation\nunification of wikis,\ntickets, shared drives] --> D
    D --> E[Conversational AI\nanswers questions\nin natural language]

    style A fill:#4A90D9,color:#fff
    style B fill:#7B68EE,color:#fff
    style C fill:#50C878,color:#fff
    style D fill:#FF8C00,color:#fff
    style E fill:#DC143C,color:#fff

RAG Operational Flow

sequenceDiagram
    participant U as User
    participant A as Application
    participant E as Embedding Model
    participant Q as Qdrant (Vector DB)
    participant L as LLM (GPT)

    U->>A: Natural language question
    A->>E: Convert question to vector
    E-->>A: Question vector
    A->>Q: Vector similarity search
    Q-->>A: Top-K most similar chunks
    A->>L: Question + retrieved context
    L-->>A: Response grounded in documents
    A-->>U: Response + cited sources

2.2 Key Advantages

graph LR
    A[Accurate\nResponses] --> B[Reduced\nHallucinations]
    B --> C[Domain\nExpertise]
    C --> A

    A1[Grounded in\norganization-specific\nknowledge base]
    B1[LLM constrained to\nretrieved context,\ncannot fabricate]
    C1[Expert behavior\nas the KB grows]

    A --- A1
    B --- B1
    C --- C1

    style A fill:#2ecc71,color:#fff
    style B fill:#3498db,color:#fff
    style C fill:#9b59b6,color:#fff

2.3 Common Challenges

mindmap
  root((RAG Challenges))
    Chunking & Noise
      Too aggressive → context loss
      Too large → noisy retrieval
      Boilerplate pollutes results
    Retrieval Quality
      Number of results to retrieve
      Result ranking
      Metadata usage
    Data Drift
      Updated policies
      Evolving systems
      Stale embeddings without reloading
    Metadata Gaps
      Less precise filtering
      Difficult to tune
      Degraded retrieval

3. Demo 1 — Basic RAG Pipeline (Clip 1)

Objective: Build an assistant on an IT knowledge base using LlamaIndex and Qdrant.

3.1 Project Structure

Clip1-v2/
├── backend/
│   ├── rag_engine.py      # RAG Engine (core of the system)
│   ├── main.py            # FastAPI application
│   └── requirements.txt
├── frontend/              # Next.js chat interface
├── data/
│   └── sample_it_kb.csv   # IT knowledge base (CSV format)
└── docker/
    └── docker-compose.yml

3.2 Docker Configuration — Qdrant

# docker-compose.yml
version: '3.8'

services:
  qdrant:
    image: qdrant/qdrant:latest
    container_name: qdrant-it-kb
    ports:
      - "6333:6333"   # REST API
      - "6334:6334"   # gRPC
    volumes:
      - qdrant_storage:/qdrant/storage
    environment:
      - QDRANT__SERVICE__GRPC_PORT=6334
    restart: unless-stopped

volumes:
  qdrant_storage:
    driver: local

Starting up:

# Launch Qdrant
cd Clip1-v2/docker
docker-compose up -d

# Verify dashboard
# http://localhost:6333/dashboard

3.3 RAG Engine — rag_engine.py

The complete RAG pipeline: loading → chunking → embeddings → storage → querying.

flowchart TD
    A[CSV\nsample_it_kb.csv] --> B[Pandas loading\n_load_it_kb_data]
    B --> C[Document creation\nwith metadata]
    C --> D[Chunking\nSentenceSplitter\n512 chars / 50 overlap]
    D --> E[Embedding generation\ntext-embedding-3-small]
    E --> F[(Qdrant\nVector Store)]
    G[User question] --> H[Question embedding]
    H --> F
    F --> I[Top-5 similar chunks]
    I --> J[GPT-3.5-turbo\nResponse synthesis]
    J --> K[Response + Sources]

    style A fill:#f9a825,color:#000
    style F fill:#1565c0,color:#fff
    style J fill:#6a1b9a,color:#fff
    style K fill:#2e7d32,color:#fff

Initialization and Configuration

"""RAG Engine using LlamaIndex and Qdrant

This module implements a complete RAG pipeline for our IT knowledge base.
Steps:
  1. Load documents from CSV
  2. Chunk into smaller pieces
  3. Create embeddings and store in Qdrant
  4. Query and retrieve relevant information
"""

import os
import logging
from typing import List, Dict, Any
import pandas as pd

from llama_index.core import (
    Document,
    VectorStoreIndex,
    Settings,
    StorageContext
)
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
from llama_index.vector_stores.qdrant import QdrantVectorStore
from qdrant_client import QdrantClient, AsyncQdrantClient

class RAGEngine:
    def __init__(
        self,
        data_path: str,
        qdrant_url: str,
        collection_name: str,
        openai_api_key: str,
        chunk_size: int = 512,       # Maximum size of each chunk
        chunk_overlap: int = 50      # Overlap to preserve context
    ):
        self.data_path = data_path
        self.qdrant_url = qdrant_url
        self.collection_name = collection_name
        self.chunk_size = chunk_size
        self.chunk_overlap = chunk_overlap

        # 1. LLM for generating final responses
        Settings.llm = OpenAI(
            model="gpt-3.5-turbo",
            api_key=openai_api_key,
            temperature=0.1  # Low temperature = more factual responses
        )

        # 2. Embedding model for semantic search
        Settings.embed_model = OpenAIEmbedding(
            model="text-embedding-3-small",
            api_key=openai_api_key
        )

        # 3. Node parser for document chunking
        Settings.node_parser = SentenceSplitter(
            chunk_size=chunk_size,
            chunk_overlap=chunk_overlap
        )

        self.client = None
        self.vector_store = None
        self.index = None
        self.query_engine = None
        self.documents = []

Pipeline Initialization

    async def initialize(self):
        """Configures the entire RAG pipeline."""
        logger.info("Initializing RAG engine...")

        # Step 1a: Load documents from CSV
        self.documents = self._load_it_kb_data()
        logger.info(f"Loaded {len(self.documents)} knowledge base items")

        # Step 1b: Connect to Qdrant
        self.client = QdrantClient(url=self.qdrant_url)
        logger.info(f"Connected to Qdrant at {self.qdrant_url}")

        # Step 1c: Create vector store
        self.vector_store = QdrantVectorStore(
            client=self.client,
            collection_name=self.collection_name
        )

        # Step 1d: Build index (chunking + embeddings + storage)
        await self._build_index()
        logger.info("RAG engine initialization complete!")

Loading and Preparing Documents

    def _load_it_kb_data(self) -> List[Document]:
        """
        Load the IT knowledge base from CSV.
        Each row becomes a LlamaIndex Document with rich text and metadata.
        """
        df = pd.read_csv(self.data_path)
        documents = []

        for idx, row in df.iterrows():
            # Build rich, searchable text
            text_parts = []

            if 'title' in row and pd.notna(row['title']):
                text_parts.append(f"Title: {row['title']}")
            if 'problem' in row and pd.notna(row['problem']):
                text_parts.append(f"Problem: {row['problem']}")
            if 'description' in row and pd.notna(row['description']):
                text_parts.append(f"Description: {row['description']}")

            # Resolution steps are the most important part!
            if 'resolution' in row and pd.notna(row['resolution']):
                text_parts.append(f"Resolution: {row['resolution']}")
            if 'steps' in row and pd.notna(row['steps']):
                text_parts.append(f"Steps: {row['steps']}")
            if 'notes' in row and pd.notna(row['notes']):
                text_parts.append(f"Notes: {row['notes']}")

            text = "\n\n".join(text_parts)

            # Metadata for filtering and source attribution
            metadata = {
                "doc_id": str(idx),
                "system": str(row.get('system', 'Unknown')),
                "category": str(row.get('category', 'General')),
                "title": str(row.get('title', f'Item {idx}'))
            }

            # Optional fields
            for field in ['priority', 'tags', 'service', 'subcategory']:
                if field in row and pd.notna(row[field]):
                    metadata[field] = str(row[field])

            doc = Document(
                text=text,
                metadata=metadata,
                id_=f"doc_{idx}"
            )
            documents.append(doc)

        return documents

Building the Vector Index

    async def _build_index(self):
        """
        Build the vector index (where the magic happens!).
        1. Chunk documents
        2. Generate embeddings
        3. Store in Qdrant
        """
        logger.info("Building vector index...")

        # Connect Qdrant vector store to LlamaIndex
        storage_context = StorageContext.from_defaults(
            vector_store=self.vector_store
        )

        # This single call performs:
        # - Chunking with SentenceSplitter
        # - Embedding creation via OpenAI
        # - Storage in Qdrant
        self.index = VectorStoreIndex.from_documents(
            self.documents,
            storage_context=storage_context,
            show_progress=True
        )

        # Create the query engine (Top-5 chunks)
        self.query_engine = self.index.as_query_engine(
            similarity_top_k=5,
            response_mode="compact"
        )

        logger.info("Vector index built successfully!")

RAG Query

    async def query(self, question: str) -> Dict[str, Any]:
        """
        This is where RAG comes into action!
        1. Embed the question
        2. Qdrant retrieves the most similar chunks
        3. LlamaIndex sends those chunks as context to GPT
        4. GPT generates a response grounded in our knowledge base
        """
        if not self.query_engine:
            raise RuntimeError("Query engine not initialized")

        # Execute the RAG query
        # Under the hood: question → embedding → similarity search → LLM
        response = self.query_engine.query(question)

        answer = str(response)

        # Extract and format source documents (crucial for transparency!)
        sources = []
        if hasattr(response, 'source_nodes'):
            for node in response.source_nodes[:3]:  # Top 3 sources
                metadata = node.node.metadata
                snippet = node.node.get_content()[:200]
                if len(node.node.get_content()) > 200:
                    snippet += "..."

                source = {
                    "title": metadata.get("title", "Unknown"),
                    "system": metadata.get("system", "Unknown"),
                    "category": metadata.get("category", "General"),
                    "snippet": snippet
                }
                sources.append(source)

        return {
            "answer": answer,
            "sources": sources
        }

3.4 FastAPI — main.py

"""
IT Knowledge Base RAG Backend
RESTful API powered by RAG for answering IT support questions.

Architecture:
  - LlamaIndex : RAG pipeline orchestration
  - Qdrant     : vector database
  - OpenAI     : embeddings and response generation
"""

from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel, ConfigDict
from contextlib import asynccontextmanager

# Pydantic models for the API contract
class QueryRequest(BaseModel):
    question: str

class Source(BaseModel):
    title: str
    system: str
    category: str
    snippet: str

class QueryResponse(BaseModel):
    answer: str
    sources: List[Source]

# Application lifecycle management
@asynccontextmanager
async def lifespan(app: FastAPI):
    """Initializes the RAG engine at startup — cost paid only once."""
    global rag_engine
    rag_engine = RAGEngine(
        data_path="../data/sample_it_kb.csv",
        qdrant_url=os.getenv("QDRANT_URL", "http://localhost:6333"),
        collection_name=os.getenv("QDRANT_COLLECTION_NAME", "it_kb"),
        openai_api_key=os.getenv("OPENAI_API_KEY")
    )
    await rag_engine.initialize()
    logger.info("RAG engine initialized successfully!")
    yield
    logger.info("Shutting down...")

app = FastAPI(
    title="IT Knowledge Base RAG API",
    description="RAG-powered API for IT knowledge base queries",
    version="1.0.0",
    lifespan=lifespan
)

# CORS for Next.js frontend
app.add_middleware(
    CORSMiddleware,
    allow_origins=["http://localhost:3000"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

@app.post("/api/query", response_model=QueryResponse)
async def query_knowledge_base(request: QueryRequest):
    """
    Main RAG endpoint.
    Steps:
    1. Receive the question
    2. Embed the question
    3. Qdrant retrieves similar chunks
    4. LlamaIndex synthesizes a response with GPT
    5. Return response + sources
    """
    if not rag_engine:
        raise HTTPException(status_code=503, detail="RAG engine not initialized")
    if not request.question.strip():
        raise HTTPException(status_code=400, detail="Question cannot be empty")

    result = await rag_engine.query(request.question)
    return QueryResponse(**result)

@app.get("/api/stats")
async def get_stats():
    """Knowledge base statistics (for debugging)."""
    stats = await rag_engine.get_stats()
    return stats

Environment Variables

# .env
OPENAI_API_KEY=sk-your-key-here
QDRANT_URL=http://localhost:6333
QDRANT_COLLECTION_NAME=it_kb

4. Building and Integrating Knowledge Bases

4.1 Metadata and Labeling

Metadata is the hidden lever behind a high-quality RAG system. It enables filtering, prioritization, and maintainability.

graph TD
    M[Knowledge Base\nMetadata]
    M --> A[owner\nDocument owner\nauthorized or not]
    M --> B[category\nVPN, Email, Onboarding\nMore precise retrieval]
    M --> C[severity\nCritical vs low priority\nInfluences reranking]
    M --> D[timestamp\nDocument freshness\nMost recent version]
    M --> E[version\nChange management\nAvoids stale guidance]

    style M fill:#e67e22,color:#fff
    style A fill:#3498db,color:#fff
    style B fill:#2ecc71,color:#fff
    style C fill:#e74c3c,color:#fff
    style D fill:#9b59b6,color:#fff
    style E fill:#1abc9c,color:#fff

4.2 Retrieval Optimization

graph LR
    A[Index Tuning\nHNSW parameters\nSpeed vs accuracy] --> R[Optimized\nRetrieval]
    B[Hybrid Retrieval\nBM25 keywords +\nVector similarity] --> R
    C[Rerankers\nCross-encoder\nPrecise re-ranking] --> R

    style A fill:#2980b9,color:#fff
    style B fill:#27ae60,color:#fff
    style C fill:#8e44ad,color:#fff
    style R fill:#c0392b,color:#fff

1. Index Tuning (HNSW parameters): controls the speed/accuracy balance in Qdrant
2. Hybrid Retrieval: combines BM25 (exact keywords) with vector search (semantic meaning)
3. Rerankers: re-order top results with a more precise cross-encoder model

5. Demo 2 — Baseline vs Optimized Comparison (Clip 2)

5.1 Compared Parameters

5.2 Improved RAG Engine — rag_engine.py (Clip 2)

The main difference from Clip 1: configuration parameters are now injected (dependency injection), allowing multiple engines to run in parallel with different configurations.

class RAGEngine:
    def __init__(
        self,
        data_path: str,
        qdrant_url: str,
        collection_name: str,
        openai_api_key: str,
        chunk_size: int = 512,
        chunk_overlap: int = 50,
        embedding_model: str = "text-embedding-3-small",  # NEW: injectable
        similarity_top_k: int = 5                          # NEW: injectable
    ):
        # Using per-instance settings (vs globals in Clip 1)
        # Allows multiple engines with different configurations
        self.llm = OpenAI(
            model="gpt-3.5-turbo",
            api_key=openai_api_key,
            temperature=0.1
        )
        self.embed_model = OpenAIEmbedding(
            model=embedding_model,   # Configurable!
            api_key=openai_api_key
        )
        self.node_parser = SentenceSplitter(
            chunk_size=chunk_size,
            chunk_overlap=chunk_overlap
        )

Building the Index with Injected Parameters

    async def _build_index(self):
        logger.info(f"Building vector index for collection '{self.collection_name}'...")
        logger.info(f"  - Chunk size: {self.chunk_size}, Overlap: {self.chunk_overlap}")
        logger.info(f"  - Embedding model: {self.embedding_model}")
        logger.info(f"  - Top-K retrieval: {self.similarity_top_k}")

        storage_context = StorageContext.from_defaults(
            vector_store=self.vector_store
        )

        self.index = VectorStoreIndex.from_documents(
            self.documents,
            storage_context=storage_context,
            embed_model=self.embed_model,          # Injected model
            transformations=[self.node_parser],     # Injected parser
            show_progress=True
        )

        self.query_engine = self.index.as_query_engine(
            llm=self.llm,
            embed_model=self.embed_model,
            similarity_top_k=self.similarity_top_k,  # Injected Top-K
            response_mode="compact"
        )

Source Deduplication (Clip 2)

    async def query(self, question: str) -> Dict[str, Any]:
        response = self.query_engine.query(question)
        answer = str(response)

        # NEW in Clip 2: deduplication by title
        sources = []
        seen_titles = set()
        chunks_retrieved = 0

        if hasattr(response, 'source_nodes'):
            chunks_retrieved = len(response.source_nodes)

            for node in response.source_nodes:
                metadata = node.node.metadata
                title = metadata.get("title", "Unknown")

                # Skip if already seen (same document, different chunk)
                if title in seen_titles:
                    continue
                seen_titles.add(title)

                snippet = node.node.get_content()[:200]
                if len(node.node.get_content()) > 200:
                    snippet += "..."

                sources.append({
                    "title": title,
                    "system": metadata.get("system", "Unknown"),
                    "category": metadata.get("category", "General"),
                    "snippet": snippet
                })

        return {
            "answer": answer,
            "sources": sources,
            "chunks_retrieved": chunks_retrieved,
            "unique_documents": len(sources)  # NEW: unique document count
        }

6. Optimization and Maintenance at Scale

6.1 Monitoring and Maintenance

flowchart LR
    subgraph Maintenance ["Continuous Maintenance"]
        A[Stale content\ndetection\nchange tracking]
        B[Re-embedding\nstrategy\nincremental reload]
        C[Collection versioning\n& migration\nside-by-side testing in Qdrant]
        D[Monitoring\nsignals\naccuracy, latency, token usage]
    end

    A --> B --> C --> D --> A

6.2 Key Production Metrics

graph TD
    M1[Precision@K\nDoes the right chunk\nappear in the Top K?] 
    M2[Latency\nResponse time\nperceived by the user]
    M3[Coverage\nIs the KB broad enough\nto cover all questions?]
    M4[Embedding Drift\nDo old embeddings\ndegrade retrieval?]
    M5[Metadata Completeness\nAre labels\nconsistent and complete?]

    M1 --> P[Production\nPerformance]
    M2 --> P
    M3 --> P
    M4 --> P
    M5 --> P

    style P fill:#c0392b,color:#fff

7. Demo 3 — Production Features (Clip 3)

7.1 The 5 Production Improvements

mindmap
  root((Clip 3 Production))
    1 Metadata Enrichment
      severity: critical vs low
      owner: responsible team
      version: v1.0 vs v2.0
      last_updated: freshness
      error_code: exact codes ERR-xxxx
    2 Hybrid Retrieval
      Vector similarity BM25 keyword
      Result fusion
      Deduplication
    3 Reranking
      Local cross-encoder
      Relevance re-ordering
      Top-N after fusion
    4 Knowledge Freshness
      Version tracking
      Document v2.0 > v1.0
      Stale content detection
    5 Operational Metrics
      Timing per step
      BM25 contribution
      Reranking applied or not

7.2 Hybrid Retrieval — BM25 + Vector Search

sequenceDiagram
    participant Q as Question
    participant VS as Vector Search\n(semantic)
    participant BM as BM25 Search\n(exact keywords)
    participant M as Merge &\nDeduplicate
    participant R as Reranker\n(cross-encoder)
    participant L as LLM

    Q->>VS: Embed + similarity search
    Q->>BM: Tokenize + keyword match
    VS-->>M: Top-K semantic chunks
    BM-->>M: Top-K BM25 chunks
    M->>R: Fused and deduplicated list
    R->>L: Top-N reranked chunks
    L-->>Q: Response grounded in sources

Why hybrid retrieval?

Vector search understands meaning but may miss exact matches. BM25 excels on precise terms like error codes or product names. Together, they form a complete strategy.

Example:
  Question: "Error code ERR-1302"
  → Vector search: returns generic error-related content
  → BM25: finds exactly "ERR-1302" in the document
  → Combined result: the right document is retrieved

7.3 Reranking with Cross-encoder

Without reranking:
  1. VPN Troubleshooting (v1.0)    ← old version
  2. VPN Overview                  ← less relevant
  3. VPN Troubleshooting (v2.0)    ← the right answer but in 3rd position

With reranking (cross-encoder):
  1. VPN Troubleshooting (v2.0)    ← most recent and relevant version
  2. VPN Troubleshooting (v1.0)
  3. VPN Overview

7.4 Optimized RAG Engine — rag_engine_optimized.py

Imports and Configuration

"""Clip 3: Optimized RAG engine with production features

Improvements:
  1. METADATA ENRICHMENT — severity, owner, version, last_updated, error_code
  2. HYBRID RETRIEVAL     — Vector similarity + BM25 keyword matching
  3. RERANKING            — Local cross-encoder (no external API)
  4. KNOWLEDGE FRESHNESS  — Version tracking via metadata
  5. OPERATIONAL METRICS  — Timing, chunk counts, hybrid indicators
"""

import time
from datetime import datetime
from rank_bm25 import BM25Okapi      # BM25: keyword ranking algorithm
from llama_index.core.postprocessor import SentenceTransformerRerank  # Local reranker

class OptimizedRAGEngine:
    def __init__(
        self,
        data_path: str,
        qdrant_url: str,
        collection_name: str,
        openai_api_key: str,
        # Clip 2 parameters (improved chunking)
        chunk_size: int = 768,
        chunk_overlap: int = 80,
        embedding_model: str = "text-embedding-3-large",
        similarity_top_k: int = 8,
        # Clip 3 parameters (new features)
        enable_hybrid: bool = True,     # BM25 + Vector
        enable_reranking: bool = True,  # Cross-encoder reranking
        bm25_top_k: int = 5,            # BM25 results to merge
        rerank_top_n: int = 5,          # Final results after reranking
    ):
        # ...configuration...

        # Reranker initialization (local model — no API key required)
        if enable_reranking:
            self.reranker = SentenceTransformerRerank(
                model="cross-encoder/ms-marco-MiniLM-L-6-v2",
                top_n=rerank_top_n
            )
        else:
            self.reranker = None

        # State variables
        self.bm25_index = None
        self.nodes = []        # Nodes for BM25
        self.last_query_metrics = {}

Enriched Metadata Extraction

    def _load_documents_with_metadata(self) -> List[Document]:
        """
        Load with rich metadata extraction.

        Why these metadata fields influence retrieval:
          - severity  : prioritization of critical incidents
          - owner     : filtering by team
          - version   : document freshness
          - error_code: exact match for BM25
        """
        df = pd.read_csv(self.data_path)
        documents = []

        for idx, row in df.iterrows():
            # Text construction (same as previous clips)
            text_parts = [...]
            text = "\n\n".join(text_parts)

            # Enriched metadata (Clip 3)
            metadata = {
                "doc_id": str(idx),
                "system": str(row.get('system', 'Unknown')),
                "category": str(row.get('category', 'General')),
                "title": str(row.get('title', f'Item {idx}')),
                # NEW Clip 3 fields:
                "severity": str(row.get('severity', 'Medium')),
                "owner": str(row.get('owner', 'Unknown')),
                "version": str(row.get('version', '1.0')),
                "last_updated": str(row.get('last_updated', '')),
                "error_code": str(row['error_code'])
                    if pd.notna(row.get('error_code', '')) else '',
                "priority": str(row.get('priority', 'Medium')),
                "tags": str(row.get('tags', '')),
            }

            doc = Document(text=text, metadata=metadata, id_=f"doc_{idx}")
            documents.append(doc)

        # Distribution statistics (operational visibility)
        severity_counts = df['severity'].value_counts().to_dict()
        version_counts = df['version'].value_counts().to_dict()
        logger.info(f"  Severity distribution: {severity_counts}")
        logger.info(f"  Version distribution: {version_counts}")

        return documents

BM25 Index Construction

    def _build_bm25_index(self):
        """
        Build the BM25 index for keyword search.

        Why hybrid retrieval is necessary:
          1. EXACT MATCHES
             - "Error code ERR-1302" → BM25 finds the exact document
             - Vector search might return generic error-related content
          2. TECHNICAL TERMINOLOGY
             - "RADIUS authentication timeout" → BM25 matches the exact term
             - If "RADIUS" is not in training data,
               vector search may miss it
          3. PRODUCT NAMES / CODES
             - Model numbers, error codes, system names
        """
        tokenized_docs = [node.get_content().lower().split()
                          for node in self.nodes]
        self.bm25_index = BM25Okapi(tokenized_docs)
        logger.info(f"  BM25 index built with {len(self.nodes)} documents")

Complete Query Pipeline (6 Steps)

    async def query(self, question: str) -> Dict[str, Any]:
        """
        Production RAG pipeline — 6 steps:
        1. Vector retrieval (semantic similarity)
        2. BM25 retrieval (keyword matching)
        3. Merge and deduplicate results
        4. Rerank by relevance
        5. Generate response with LLM
        6. Collect operational metrics
        """
        start_time = time.time()
        metrics = {"query": question, "timestamp": datetime.now().isoformat()}

        # --- STEP 1: Vector Retrieval ---
        vector_start = time.time()
        vector_retriever = self.index.as_retriever(similarity_top_k=self.similarity_top_k)
        vector_results = vector_retriever.retrieve(question)
        vector_time = time.time() - vector_start
        metrics["vector_retrieval_ms"] = round(vector_time * 1000, 2)

        # --- STEP 2: BM25 Retrieval ---
        bm25_results = []
        if self.enable_hybrid:
            bm25_start = time.time()
            bm25_results = self._bm25_search(question, self.bm25_top_k)
            bm25_time = time.time() - bm25_start
            metrics["bm25_retrieval_ms"] = round(bm25_time * 1000, 2)

        # --- STEP 3: Hybrid Merge ---
        merged_results = []
        seen_ids = set()

        # Vector results first (calibrated similarity scores)
        for node_with_score in vector_results:
            node_id = node_with_score.node.node_id
            if node_id not in seen_ids:
                seen_ids.add(node_id)
                merged_results.append(node_with_score)

        # Add unique BM25 results
        bm25_unique_count = 0
        for node_with_score in bm25_results:
            node_id = node_with_score.node.node_id
            if node_id not in seen_ids:
                seen_ids.add(node_id)
                merged_results.append(node_with_score)
                bm25_unique_count += 1

        metrics["hybrid_unique_from_bm25"] = bm25_unique_count

        # --- STEP 4: Reranking ---
        final_results = merged_results
        if self.enable_reranking and self.reranker and len(merged_results) > 0:
            rerank_start = time.time()
            final_results = self.reranker.postprocess_nodes(
                merged_results,
                query_str=question
            )
            rerank_time = time.time() - rerank_start
            metrics["rerank_time_ms"] = round(rerank_time * 1000, 2)
            metrics["rerank_applied"] = True

        # --- STEP 5: LLM Generation ---
        context_parts = []
        for i, node_with_score in enumerate(final_results):
            node = node_with_score.node
            metadata = node.metadata
            context_parts.append(
                f"[Source {i+1}]\n"
                f"Title: {metadata.get('title', 'Unknown')}\n"
                f"Severity: {metadata.get('severity', 'Unknown')}\n"
                f"Version: {metadata.get('version', '1.0')}\n"
                f"Content: {node.get_content()[:500]}"
            )

        context = "\n\n---\n\n".join(context_parts)
        prompt = f"""You are an IT support expert.
Answer the user's question based solely on the provided context.
Prioritize information from high-severity and recent-version documents.

Context:
{context}

Question: {question}
Answer:"""

        response = self.llm.complete(prompt)
        answer = str(response)

        # --- STEP 6: Format sources with rich metadata ---
        sources = []
        seen_titles = set()
        for i, node_with_score in enumerate(final_results):
            node = node_with_score.node
            metadata = node.metadata
            title = metadata.get("title", "Unknown")

            if title in seen_titles:
                continue
            seen_titles.add(title)

            sources.append({
                "title": title,
                "system": metadata.get("system", "Unknown"),
                "category": metadata.get("category", "General"),
                "snippet": node.get_content()[:250] + "...",
                "severity": metadata.get("severity"),
                "owner": metadata.get("owner"),
                "version": metadata.get("version"),
                "last_updated": metadata.get("last_updated"),
                "error_code": metadata.get("error_code"),
                "rerank_position": i + 1
            })

        # Total metrics
        metrics["total_time_ms"] = round((time.time() - start_time) * 1000, 2)
        self.last_query_metrics = metrics

        return {
            "answer": answer,
            "sources": sources,
            "metrics": metrics
        }

BM25 Search

    def _bm25_search(self, query: str, top_k: int) -> List[NodeWithScore]:
        """
        BM25 keyword search.
        Returns nodes with normalized BM25 scores in [0, 1].
        """
        if self.bm25_index is None:
            return []

        # Tokenize the question
        query_tokens = query.lower().split()

        # BM25 scores for all documents
        scores = self.bm25_index.get_scores(query_tokens)

        # Top-K indices
        top_indices = sorted(range(len(scores)),
                             key=lambda i: scores[i], reverse=True)[:top_k]

        results = []
        max_score = max(scores) if max(scores) > 0 else 1.0

        for idx in top_indices:
            if scores[idx] > 0:
                normalized_score = scores[idx] / max_score
                results.append(NodeWithScore(
                    node=self.nodes[idx],
                    score=normalized_score
                ))

        return results

7.5 Extended FastAPI — main.py (Clip 3)

# Two engines initialized at startup for comparison
rag_engine_baseline: Optional[RAGEngine] = None    # Same as Clip 1
rag_engine_optimized: Optional[OptimizedRAGEngine] = None  # Clip 3

# Separate Qdrant collections
QDRANT_COLLECTION_BASELINE  = "it_kb_baseline"
QDRANT_COLLECTION_OPTIMIZED = "it_kb_optimized_v3"

@asynccontextmanager
async def lifespan(app: FastAPI):
    # Baseline engine (UNCHANGED from Clip 1 — control group)
    rag_engine_baseline = RAGEngine(
        data_path="../data/diverse_it_kb.csv",
        qdrant_url=os.getenv("QDRANT_URL", "http://localhost:6333"),
        collection_name=QDRANT_COLLECTION_BASELINE,
        openai_api_key=os.getenv("OPENAI_API_KEY"),
        chunk_size=512,
        chunk_overlap=50,
        embedding_model="text-embedding-3-small",
        similarity_top_k=5
    )
    await rag_engine_baseline.initialize()

    # Optimized engine (all production features)
    rag_engine_optimized = OptimizedRAGEngine(
        data_path="../data/diverse_it_kb.csv",
        qdrant_url=os.getenv("QDRANT_URL", "http://localhost:6333"),
        collection_name=QDRANT_COLLECTION_OPTIMIZED,
        openai_api_key=os.getenv("OPENAI_API_KEY"),
        chunk_size=768,
        chunk_overlap=80,
        embedding_model="text-embedding-3-large",
        similarity_top_k=8,
        enable_hybrid=True,
        enable_reranking=True,
        bm25_top_k=5,
        rerank_top_n=5,
    )
    await rag_engine_optimized.initialize()
    yield

@app.post("/api/compare", response_model=CompareResponse)
async def compare_engines(request: CompareRequest):
    """
    Compare baseline vs optimized engine.

    Demo scenarios:
      1. "Error code ERR-1302 VPN"
         → Hybrid retrieval catches the exact error code
      2. "How do I escalate a severe VPN outage?"
         → Reranking promotes Critical severity documents
      3. "Legacy SSO migration steps"
         → BM25 matches "Legacy SSO" as an exact phrase
      4. "VPN troubleshooting guide"
         → Optimized returns v2.0 (Jan 2025), baseline may return v1.0
    """
    # Parallel execution for comparison
    baseline_result = await rag_engine_baseline.query(request.question)
    optimized_result = await rag_engine_optimized.query(request.question)

    return CompareResponse(
        baseline=QueryResponse(**baseline_result),
        optimized=QueryResponse(**optimized_result),
        baseline_config=EngineConfig(
            chunk_size=512, chunk_overlap=50,
            embedding_model="text-embedding-3-small",
            top_k=5, hybrid_enabled=False, reranking_enabled=False
        ),
        optimized_config=EngineConfig(
            chunk_size=768, chunk_overlap=80,
            embedding_model="text-embedding-3-large",
            top_k=8, hybrid_enabled=True, reranking_enabled=True
        ),
    )

Enriched Pydantic Models (Clip 3)

class Source(BaseModel):
    """Source document information"""
    title: str
    system: str
    category: str
    snippet: str
    # Clip 3: rich metadata fields
    severity: Optional[str] = None
    owner: Optional[str] = None
    version: Optional[str] = None
    last_updated: Optional[str] = None
    error_code: Optional[str] = None
    rerank_position: Optional[int] = None  # Position after reranking

class Metrics(BaseModel):
    """Operational metrics for RAG health monitoring"""
    total_time_ms: Optional[float] = None
    vector_retrieval_ms: Optional[float] = None
    bm25_retrieval_ms: Optional[float] = None
    rerank_time_ms: Optional[float] = None
    llm_generation_ms: Optional[float] = None
    chunks_retrieved: Optional[int] = None
    hybrid_unique_from_bm25: Optional[int] = None  # BM25 contribution
    rerank_applied: Optional[bool] = None
    hybrid_enabled: Optional[bool] = None

8. Three-Clip Comparison Table

graph LR
    subgraph C1["Clip 1 — Baseline"]
        direction TB
        C1A[chunk_size: 512]
        C1B[overlap: 50]
        C1C[text-embedding-3-small]
        C1D[top_k: 5]
        C1E[vector search only]
        C1F[no reranking]
    end

    subgraph C2["Clip 2 — Optimized"]
        direction TB
        C2A[chunk_size: 768]
        C2B[overlap: 80]
        C2C[text-embedding-3-large]
        C2D[top_k: 8]
        C2E[vector search only]
        C2F[no reranking]
    end

    subgraph C3["Clip 3 — Production"]
        direction TB
        C3A[chunk_size: 768]
        C3B[overlap: 80]
        C3C[text-embedding-3-large]
        C3D[top_k: 8]
        C3E[Hybrid BM25 + Vector]
        C3F[Cross-encoder reranking]
        C3G[Rich metadata]
        C3H[Operational metrics]
    end

    C1 --> C2 --> C3

9. Installation and Quick Start

Prerequisites

• Python 3.11 or 3.12 (⚠️ not 3.13 — LlamaIndex incompatibility)
• Node.js 18+
• Docker Desktop (for Qdrant)
• OpenAI API Key (get a key)

Getting Started

# 1. Launch Qdrant (vector database)
cd Clip1-v2/docker
docker-compose up -d

# Verify: http://localhost:6333/dashboard

# 2. Configure the Python backend
cd Clip1-v2/backend

# Create virtual environment
python -m venv venv

# Activate (Windows PowerShell)
.\venv\Scripts\Activate.ps1

# Install dependencies
pip install -r requirements.txt

# Create .env file
echo "OPENAI_API_KEY=sk-your-key-here" > .env

# Start the server (wait 1-2 min for initialization)
python main.py
# Expected message: "RAG engine initialization complete!"

# 3. Configure the Next.js frontend
cd Clip1-v2/frontend
npm install
npm run dev

# 4. Open the application
# http://localhost:3000

API Testing

# Health check
curl http://localhost:8000/

# RAG query
curl -X POST http://localhost:8000/api/query \
  -H "Content-Type: application/json" \
  -d '{"question": "How do I fix VPN connection timeouts?"}'

# Statistics
curl http://localhost:8000/api/stats

# Baseline vs optimized comparison (Clip 3)
curl -X POST http://localhost:8000/api/compare \
  -H "Content-Type: application/json" \
  -d '{"question": "Error code ERR-1302 VPN connection refused"}'

Common Troubleshooting

10. Key Demonstrated Concepts

Semantic Search vs Keyword Search

Semantic Search (Vector):
  Question: "My computer is running slow"
  → Finds: "Device performance issues", "Laptop running slowly"
  → Understands MEANING, not exact words

Keyword Search (BM25):
  Question: "Error ERR-1302"
  → Finds: documents containing exactly "ERR-1302"
  → EXACT term matching

Chunking Strategy

Too small (e.g., 128 chars):
  + High granularity
  - Loses context
  - "Restart the service" without knowing WHICH service

Optimal (512-768 chars with overlap):
  + Preserves context
  + Sufficient granularity
  - Overlap (50-80 chars) avoids cutting important ideas

Too large (e.g., 2048 chars):
  + Maximum context
  - Less precise retrieval (too much noise in each chunk)

Grounding vs Hallucination

Without RAG (LLM alone):
  Question: "How do I configure our company's VPN?"
  Response: Generic instructions based on training data
  → May be incorrect or not reflect your config

With RAG (LLM + KB):
  Question: "How do I configure our company's VPN?"
  Retrieval: Your internal VPN guide with your specific settings
  Response: Instructions grounded in YOUR documentation
  → Accurate, verifiable, with cited sources

Complete Final Architecture (Clip 3)

flowchart TD
    UI[Chat Interface\nNext.js :3000]
    API[FastAPI :8000\n/api/query\n/api/compare\n/api/stats]

    subgraph RAG_Pipeline["Optimized RAG Pipeline"]
        VR[Vector Retrieval\ntext-embedding-3-large\nTop-8 similarity]
        BR[BM25 Retrieval\nKeyword matching\nTop-5 BM25]
        MG[Merge & Deduplicate\nResult fusion]
        RK[Reranking\ncross-encoder/ms-marco\nTop-5 final]
        GN[LLM Generation\nGPT-3.5-turbo / GPT-4o\nWith context]
    end

    subgraph Storage["Storage"]
        QD[(Qdrant\nVector DB\n:6333)]
        CSV[CSV\ndiverse_it_kb.csv\nwith metadata]
    end

    UI --> API
    API --> VR
    API --> BR
    VR --> MG
    BR --> MG
    MG --> RK
    RK --> GN
    GN --> API
    VR --> QD
    CSV --> QD

    style UI fill:#2980b9,color:#fff
    style API fill:#27ae60,color:#fff
    style QD fill:#8e44ad,color:#fff
    style GN fill:#e74c3c,color:#fff

11. Additional Resources

Official Documentation

• LlamaIndex Documentation
• Qdrant Documentation
• OpenAI Embeddings Guide
• FastAPI Documentation
• Next.js Documentation

Recommended Learning Path

graph TD
    B[Beginner\nClip 1 — Basic Pipeline\nRead rag_engine.py\nExperiment with different questions]
    I[Intermediate\nClip 2 — Comparison\nAdjust chunk sizes and embeddings\nAdd metadata]
    A[Advanced\nClip 3 — Production\nHybrid search and reranking\nMonitoring and metrics]

    B --> I --> A

Extension Ideas

• Add conversation memory (chat history)
• Implement response streaming
• Create evaluation benchmarks
• Production deployment (AWS, GCP, Azure)
• Integration with other vector databases (Pinecone, Weaviate)
• Multimodal support (images, PDFs)

Search Terms

integrating · knowledge · bases · rags · rag · vector · search · embeddings · artificial · intelligence · generative · ai · clip · bm25 · configuration · engine · index · pipeline · production · architecture · comparison · enriched · fastapi · initialization