Lesson 4.5: Practice - Q&A on Documents

Duration: 90 minutes

Learning Objectives

By the end of this lesson, you will be able to:

1. Build a complete RAG pipeline from documents to answers
2. Load and process documents from various sources
3. Implement the full indexing and query flow
4. Add source attribution to responses
5. Handle edge cases and improve answer quality

Project Overview

In this lesson, you will build a document Q&A system that can:

• Load documents from a folder
• Chunk and embed documents
• Store embeddings in a vector database
• Answer questions using retrieved context
• Show sources for each answer

We will build this step by step, using Chroma for simplicity.

Project Setup

Create a new project:

mkdir rag-qa-demo
cd rag-qa-demo
npm init -y

Install dependencies:

npm install typescript tsx @types/node openai chromadb dotenv

Create tsconfig.json:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "outDir": "dist"
  },
  "include": ["src/**/*"]
}

Create .env file:

OPENAI_API_KEY=sk-proj-your-openai-key-here

Create project structure:

mkdir -p src/lib
mkdir -p src/docs
touch src/index.ts
touch src/lib/document-loader.ts
touch src/lib/chunker.ts
touch src/lib/vector-store.ts
touch src/lib/qa-chain.ts

Step 1: Document Loader

Create src/lib/document-loader.ts:

import * as fs from 'fs';
import * as path from 'path';

export interface Document {
  id: string;
  content: string;
  metadata: {
    source: string;
    filename: string;
    type: string;
  };
}

export class DocumentLoader {
  private supportedExtensions = ['.txt', '.md', '.json'];

  async loadFromDirectory(dirPath: string): Promise<Document[]> {
    const documents: Document[] = [];
    const files = fs.readdirSync(dirPath);

    for (const file of files) {
      const filePath = path.join(dirPath, file);
      const stat = fs.statSync(filePath);

      if (stat.isFile()) {
        const ext = path.extname(file).toLowerCase();

        if (this.supportedExtensions.includes(ext)) {
          const doc = await this.loadFile(filePath);
          if (doc) {
            documents.push(doc);
          }
        }
      }
    }

    console.log(`Loaded ${documents.length} documents from ${dirPath}`);
    return documents;
  }

  async loadFile(filePath: string): Promise<Document | null> {
    const ext = path.extname(filePath).toLowerCase();
    const filename = path.basename(filePath);
    const content = fs.readFileSync(filePath, 'utf-8');

    if (!content.trim()) {
      console.log(`Skipping empty file: ${filename}`);
      return null;
    }

    let processedContent: string;

    switch (ext) {
      case '.json':
        processedContent = this.processJson(content);
        break;
      case '.md':
        processedContent = this.processMarkdown(content);
        break;
      default:
        processedContent = content;
    }

    return {
      id: this.generateId(filePath),
      content: processedContent,
      metadata: {
        source: filePath,
        filename,
        type: ext.replace('.', ''),
      },
    };
  }

  private processJson(content: string): string {
    try {
      const data = JSON.parse(content);
      return JSON.stringify(data, null, 2);
    } catch {
      return content;
    }
  }

  private processMarkdown(content: string): string {
    // Keep markdown as-is, it embeds well
    return content;
  }

  private generateId(filePath: string): string {
    // Create a deterministic ID from the file path
    return Buffer.from(filePath).toString('base64').replace(/[/+=]/g, '_');
  }
}

Step 2: Text Chunker

Create src/lib/chunker.ts:

export interface Chunk {
  id: string;
  content: string;
  metadata: {
    source: string;
    filename: string;
    chunkIndex: number;
    totalChunks?: number;
  };
}

export interface ChunkingOptions {
  chunkSize: number;
  chunkOverlap: number;
}

export class TextChunker {
  private options: ChunkingOptions;
  private separators = ['\n\n', '\n', '. ', ', ', ' '];

  constructor(options: ChunkingOptions = { chunkSize: 1000, chunkOverlap: 200 }) {
    this.options = options;
  }

  chunkDocuments(
    documents: { id: string; content: string; metadata: Record<string, string> }[]
  ): Chunk[] {
    const allChunks: Chunk[] = [];

    for (const doc of documents) {
      const chunks = this.chunkText(doc.content);
      const docChunks = chunks.map((content, index) => ({
        id: `${doc.id}_chunk_${index}`,
        content,
        metadata: {
          source: doc.metadata.source || '',
          filename: doc.metadata.filename || '',
          chunkIndex: index,
          totalChunks: chunks.length,
        },
      }));

      allChunks.push(...docChunks);
    }

    console.log(`Created ${allChunks.length} chunks from ${documents.length} documents`);
    return allChunks;
  }

  private chunkText(text: string): string[] {
    return this.recursiveSplit(text, 0);
  }

  private recursiveSplit(text: string, separatorIndex: number): string[] {
    const { chunkSize, chunkOverlap } = this.options;

    // Base case: text is small enough
    if (text.length <= chunkSize) {
      return text.trim() ? [text.trim()] : [];
    }

    // Base case: no more separators
    if (separatorIndex >= this.separators.length) {
      return this.hardSplit(text, chunkSize, chunkOverlap);
    }

    const separator = this.separators[separatorIndex];
    const parts = text.split(separator);

    // If separator didn't help, try next one
    if (parts.length === 1) {
      return this.recursiveSplit(text, separatorIndex + 1);
    }

    const chunks: string[] = [];
    let currentChunk = '';

    for (const part of parts) {
      const trimmedPart = part.trim();
      if (!trimmedPart) continue;

      const testChunk = currentChunk ? currentChunk + separator + trimmedPart : trimmedPart;

      if (testChunk.length <= chunkSize) {
        currentChunk = testChunk;
      } else {
        // Save current chunk if it exists
        if (currentChunk.trim()) {
          chunks.push(currentChunk.trim());
        }

        // Handle part that might be too large
        if (trimmedPart.length > chunkSize) {
          const subChunks = this.recursiveSplit(trimmedPart, separatorIndex + 1);
          chunks.push(...subChunks);
          currentChunk = '';
        } else {
          currentChunk = trimmedPart;
        }
      }
    }

    // Don't forget the last chunk
    if (currentChunk.trim()) {
      chunks.push(currentChunk.trim());
    }

    // Add overlap between chunks
    return this.addOverlap(chunks, chunkOverlap);
  }

  private hardSplit(text: string, size: number, overlap: number): string[] {
    const chunks: string[] = [];
    let start = 0;

    while (start < text.length) {
      const end = Math.min(start + size, text.length);
      const chunk = text.slice(start, end).trim();
      if (chunk) {
        chunks.push(chunk);
      }
      start += size - overlap;
    }

    return chunks;
  }

  private addOverlap(chunks: string[], overlapSize: number): string[] {
    if (chunks.length <= 1 || overlapSize === 0) {
      return chunks;
    }

    const result: string[] = [chunks[0]];

    for (let i = 1; i < chunks.length; i++) {
      const prevChunk = chunks[i - 1];
      const overlap = prevChunk.slice(-overlapSize);
      result.push(overlap + ' ' + chunks[i]);
    }

    return result;
  }
}

Step 3: Vector Store

Create src/lib/vector-store.ts:

import { ChromaClient, Collection } from 'chromadb';

import type { Chunk } from './chunker.js';

export interface SearchResult {
  id: string;
  content: string;
  score: number;
  metadata: {
    source: string;
    filename: string;
  };
}

export class VectorStore {
  private client: ChromaClient;
  private collection: Collection | null = null;
  private collectionName: string;

  constructor(collectionName: string = 'documents') {
    this.client = new ChromaClient();
    this.collectionName = collectionName;
  }

  async initialize(): Promise<void> {
    this.collection = await this.client.getOrCreateCollection({
      name: this.collectionName,
      metadata: { 'hnsw:space': 'cosine' },
    });

    const count = await this.collection.count();
    console.log(
      `Vector store initialized. Collection "${this.collectionName}" has ${count} documents.`
    );
  }

  async addChunks(chunks: Chunk[]): Promise<void> {
    if (!this.collection) {
      throw new Error('Vector store not initialized');
    }

    if (chunks.length === 0) {
      console.log('No chunks to add');
      return;
    }

    // Chroma handles embedding automatically with its default embedding function
    await this.collection.add({
      ids: chunks.map((c) => c.id),
      documents: chunks.map((c) => c.content),
      metadatas: chunks.map((c) => ({
        source: c.metadata.source,
        filename: c.metadata.filename,
        chunkIndex: String(c.metadata.chunkIndex),
      })),
    });

    console.log(`Added ${chunks.length} chunks to vector store`);
  }

  async search(query: string, limit: number = 5): Promise<SearchResult[]> {
    if (!this.collection) {
      throw new Error('Vector store not initialized');
    }

    const results = await this.collection.query({
      queryTexts: [query],
      nResults: limit,
    });

    return results.ids[0].map((id, index) => ({
      id,
      content: results.documents[0][index] || '',
      score: 1 - (results.distances?.[0][index] || 0), // Convert distance to similarity
      metadata: {
        source: (results.metadatas?.[0][index]?.source as string) || '',
        filename: (results.metadatas?.[0][index]?.filename as string) || '',
      },
    }));
  }

  async clear(): Promise<void> {
    await this.client.deleteCollection({ name: this.collectionName });
    await this.initialize();
    console.log('Vector store cleared');
  }

  async getCount(): Promise<number> {
    if (!this.collection) {
      throw new Error('Vector store not initialized');
    }
    return await this.collection.count();
  }
}

Step 4: QA Chain

Create src/lib/qa-chain.ts:

import OpenAI from 'openai';

import type { SearchResult } from './vector-store.js';

export interface QAResponse {
  answer: string;
  sources: {
    filename: string;
    content: string;
    score: number;
  }[];
}

export class QAChain {
  private openai: OpenAI;
  private model: string;

  constructor(model: string = 'gpt-4o-mini') {
    this.openai = new OpenAI();
    this.model = model;
  }

  async answer(question: string, context: SearchResult[]): Promise<QAResponse> {
    // Build context string from search results
    const contextText = this.buildContext(context);

    // Create prompt
    const systemPrompt = `You are a helpful assistant that answers questions based on the provided context.

Rules:
1. Only use information from the provided context to answer questions.
2. If the context doesn't contain enough information to answer, say "I don't have enough information to answer this question."
3. Be concise and direct in your answers.
4. If you quote from the context, indicate which source it came from.`;

    const userPrompt = `Context:
${contextText}

Question: ${question}

Please answer the question based only on the context provided above.`;

    // Call OpenAI
    const response = await this.openai.chat.completions.create({
      model: this.model,
      messages: [
        { role: 'system', content: systemPrompt },
        { role: 'user', content: userPrompt },
      ],
      temperature: 0.3, // Lower temperature for more factual responses
      max_tokens: 1000,
    });

    const answer = response.choices[0]?.message?.content || 'Unable to generate answer';

    return {
      answer,
      sources: context.map((c) => ({
        filename: c.metadata.filename,
        content: c.content.substring(0, 200) + '...',
        score: c.score,
      })),
    };
  }

  private buildContext(results: SearchResult[]): string {
    return results
      .map(
        (result, index) =>
          `[Source ${index + 1}: ${result.metadata.filename}]
${result.content}
`
      )
      .join('\n---\n\n');
  }

  async answerWithStreaming(
    question: string,
    context: SearchResult[],
    onToken: (token: string) => void
  ): Promise<QAResponse> {
    const contextText = this.buildContext(context);

    const systemPrompt = `You are a helpful assistant that answers questions based on the provided context.

Rules:
1. Only use information from the provided context to answer questions.
2. If the context doesn't contain enough information to answer, say "I don't have enough information to answer this question."
3. Be concise and direct in your answers.`;

    const userPrompt = `Context:
${contextText}

Question: ${question}

Please answer the question based only on the context provided above.`;

    const stream = await this.openai.chat.completions.create({
      model: this.model,
      messages: [
        { role: 'system', content: systemPrompt },
        { role: 'user', content: userPrompt },
      ],
      temperature: 0.3,
      max_tokens: 1000,
      stream: true,
    });

    let fullAnswer = '';

    for await (const chunk of stream) {
      const token = chunk.choices[0]?.delta?.content || '';
      if (token) {
        fullAnswer += token;
        onToken(token);
      }
    }

    return {
      answer: fullAnswer,
      sources: context.map((c) => ({
        filename: c.metadata.filename,
        content: c.content.substring(0, 200) + '...',
        score: c.score,
      })),
    };
  }
}

Step 5: Main Application

Create src/index.ts:

import 'dotenv/config';
import * as readline from 'readline';

import { TextChunker } from './lib/chunker.js';
import { DocumentLoader } from './lib/document-loader.js';
import { QAChain } from './lib/qa-chain.js';
import { VectorStore } from './lib/vector-store.js';

class DocumentQA {
  private loader: DocumentLoader;
  private chunker: TextChunker;
  private vectorStore: VectorStore;
  private qaChain: QAChain;
  private isInitialized = false;

  constructor() {
    this.loader = new DocumentLoader();
    this.chunker = new TextChunker({
      chunkSize: 1000,
      chunkOverlap: 200,
    });
    this.vectorStore = new VectorStore('qa_documents');
    this.qaChain = new QAChain('gpt-4o-mini');
  }

  async initialize(): Promise<void> {
    await this.vectorStore.initialize();
    this.isInitialized = true;
  }

  async loadDocuments(dirPath: string): Promise<void> {
    console.log(`\nLoading documents from: ${dirPath}`);

    // Load documents
    const documents = await this.loader.loadFromDirectory(dirPath);

    if (documents.length === 0) {
      console.log('No documents found in directory');
      return;
    }

    // Chunk documents
    const chunks = this.chunker.chunkDocuments(
      documents.map((d) => ({
        id: d.id,
        content: d.content,
        metadata: d.metadata,
      }))
    );

    // Clear existing data and add new chunks
    await this.vectorStore.clear();
    await this.vectorStore.addChunks(chunks);

    console.log(`\nDocuments indexed successfully!`);
    console.log(`- Documents loaded: ${documents.length}`);
    console.log(`- Chunks created: ${chunks.length}`);
  }

  async askQuestion(question: string): Promise<void> {
    if (!this.isInitialized) {
      throw new Error('System not initialized. Call initialize() first.');
    }

    const count = await this.vectorStore.getCount();
    if (count === 0) {
      console.log('\nNo documents indexed. Load documents first.');
      return;
    }

    console.log('\nSearching for relevant context...');

    // Search for relevant chunks
    const results = await this.vectorStore.search(question, 5);

    if (results.length === 0) {
      console.log('No relevant documents found.');
      return;
    }

    console.log(`Found ${results.length} relevant chunks.\n`);
    console.log('Answer:');
    console.log('-'.repeat(50));

    // Generate answer with streaming
    const response = await this.qaChain.answerWithStreaming(question, results, (token) =>
      process.stdout.write(token)
    );

    console.log('\n' + '-'.repeat(50));

    // Show sources
    console.log('\nSources:');
    const uniqueSources = [...new Set(response.sources.map((s) => s.filename))];
    uniqueSources.forEach((source, i) => {
      const sourceInfo = response.sources.find((s) => s.filename === source);
      console.log(
        `${i + 1}. ${source} (relevance: ${((sourceInfo?.score || 0) * 100).toFixed(1)}%)`
      );
    });
  }
}

// Interactive CLI
async function main() {
  const qa = new DocumentQA();
  await qa.initialize();

  const rl = readline.createInterface({
    input: process.stdin,
    output: process.stdout,
  });

  console.log('\n' + '='.repeat(60));
  console.log('Document Q&A System');
  console.log('='.repeat(60));
  console.log('\nCommands:');
  console.log('  load <path>  - Load documents from a directory');
  console.log('  ask <question> - Ask a question about the documents');
  console.log('  quit - Exit the program');
  console.log('');

  const prompt = () => {
    rl.question('\n> ', async (input) => {
      const trimmedInput = input.trim();

      if (!trimmedInput) {
        prompt();
        return;
      }

      if (trimmedInput === 'quit' || trimmedInput === 'exit') {
        console.log('Goodbye!');
        rl.close();
        process.exit(0);
      }

      if (trimmedInput.startsWith('load ')) {
        const path = trimmedInput.slice(5).trim();
        try {
          await qa.loadDocuments(path);
        } catch (error) {
          console.error('Error loading documents:', error);
        }
      } else if (trimmedInput.startsWith('ask ')) {
        const question = trimmedInput.slice(4).trim();
        try {
          await qa.askQuestion(question);
        } catch (error) {
          console.error('Error answering question:', error);
        }
      } else {
        // Treat as a question
        try {
          await qa.askQuestion(trimmedInput);
        } catch (error) {
          console.error('Error:', error);
        }
      }

      prompt();
    });
  };

  prompt();
}

main().catch(console.error);

Step 6: Create Sample Documents

Create some sample documents to test with. Create src/docs/product-guide.md:

# Product User Guide

## Getting Started

Welcome to our product! This guide will help you get started quickly.

### System Requirements

- Operating System: Windows 10+, macOS 10.15+, or Linux
- Memory: 4GB RAM minimum, 8GB recommended
- Storage: 500MB free disk space
- Internet connection required for activation

### Installation

1. Download the installer from our website
2. Run the installer and follow the prompts
3. Enter your license key when prompted
4. Restart your computer to complete installation

## Account Management

### Creating an Account

To create a new account:

1. Click "Sign Up" on the login page
2. Enter your email address
3. Create a strong password (minimum 8 characters)
4. Verify your email by clicking the link we send you

### Password Reset

If you forgot your password:

1. Go to the login page
2. Click "Forgot Password"
3. Enter your email address
4. Check your email for reset instructions
5. Click the reset link within 24 hours
6. Create a new password

### Two-Factor Authentication

We strongly recommend enabling two-factor authentication:

1. Go to Settings > Security
2. Click "Enable 2FA"
3. Scan the QR code with your authenticator app
4. Enter the verification code
5. Save your backup codes in a secure location

## Billing and Subscriptions

### Subscription Plans

We offer three subscription tiers:

**Basic Plan - $9.99/month**

- 5 projects
- 10GB storage
- Email support

**Professional Plan - $29.99/month**

- Unlimited projects
- 100GB storage
- Priority support
- Advanced analytics

**Enterprise Plan - Custom pricing**

- Everything in Professional
- Dedicated support
- Custom integrations
- SLA guarantees

### Cancellation Policy

You can cancel your subscription at any time:

1. Go to Settings > Billing
2. Click "Cancel Subscription"
3. Your access continues until the end of the billing period
4. No refunds for partial months

### Refund Policy

We offer refunds within 30 days of purchase for annual plans. Monthly plans are non-refundable but you can cancel anytime.

Create src/docs/faq.md:

# Frequently Asked Questions

## General Questions

### What is this product?

Our product is a comprehensive project management solution designed for teams of all sizes. It helps you organize tasks, collaborate with team members, and track progress efficiently.

### Is there a free trial?

Yes! We offer a 14-day free trial with full access to all Professional features. No credit card required.

### What platforms do you support?

We support:

- Web browsers (Chrome, Firefox, Safari, Edge)
- Desktop apps for Windows and macOS
- Mobile apps for iOS and Android

## Technical Questions

### How do I export my data?

To export your data:

1. Go to Settings > Data Management
2. Click "Export Data"
3. Choose your format (JSON, CSV, or PDF)
4. Download will start automatically

### What file formats can I upload?

We support the following file formats:

- Documents: PDF, DOC, DOCX, TXT
- Images: PNG, JPG, GIF, SVG
- Spreadsheets: XLS, XLSX, CSV
- Maximum file size: 100MB

### Is my data secure?

Yes, security is our top priority:

- All data is encrypted in transit (TLS 1.3)
- Data at rest is encrypted (AES-256)
- We are SOC 2 Type II certified
- Regular security audits
- GDPR compliant

## Billing Questions

### How do I update my payment method?

To update your payment method:

1. Go to Settings > Billing
2. Click "Payment Methods"
3. Add a new card or select existing one
4. Set as default payment method

### Do you offer discounts for non-profits?

Yes! Non-profit organizations get 50% off any plan. Contact our sales team with proof of non-profit status.

### What happens if my payment fails?

If a payment fails:

1. We'll notify you by email
2. You have 7 days to update payment info
3. After 7 days, account is downgraded to Basic
4. Your data is preserved for 30 days

Running the Application

1. Start the application:

npx tsx src/index.ts

2. Load your documents:

> load src/docs

3. Ask questions:

> How do I reset my password?
> What subscription plans are available?
> Is my data secure?
> How can I enable two-factor authentication?

Example Session

============================================================
Document Q&A System
============================================================

Commands:
  load <path>  - Load documents from a directory
  ask <question> - Ask a question about the documents
  quit - Exit the program

> load src/docs

Loading documents from: src/docs
Loaded 2 documents from src/docs
Created 8 chunks from 2 documents

Documents indexed successfully!
- Documents loaded: 2
- Chunks created: 8

> How do I reset my password?

Searching for relevant context...
Found 5 relevant chunks.

Answer:
--------------------------------------------------
To reset your password, follow these steps:

1. Go to the login page
2. Click "Forgot Password"
3. Enter your email address
4. Check your email for reset instructions
5. Click the reset link within 24 hours
6. Create a new password

Make sure to click the reset link within 24 hours, or you'll need to request a new one.
--------------------------------------------------

Sources:
1. product-guide.md (relevance: 87.3%)
2. faq.md (relevance: 72.1%)

> What are the subscription prices?

Searching for relevant context...
Found 5 relevant chunks.

Answer:
--------------------------------------------------
We offer three subscription tiers:

1. **Basic Plan** - $9.99/month
   - 5 projects
   - 10GB storage
   - Email support

2. **Professional Plan** - $29.99/month
   - Unlimited projects
   - 100GB storage
   - Priority support
   - Advanced analytics

3. **Enterprise Plan** - Custom pricing
   - Everything in Professional
   - Dedicated support
   - Custom integrations
   - SLA guarantees
--------------------------------------------------

Sources:
1. product-guide.md (relevance: 91.2%)

Improvements to Consider

1. Better Relevance Filtering

// Filter out low-relevance results
const filteredResults = results.filter((r) => r.score > 0.5);

2. Hybrid Search

Combine vector search with keyword search for better results:

async hybridSearch(query: string, limit: number) {
  // Vector search
  const vectorResults = await this.vectorStore.search(query, limit * 2);

  // Keyword search (simple implementation)
  const keywords = query.toLowerCase().split(' ');
  const keywordResults = vectorResults.filter(r =>
    keywords.some(kw => r.content.toLowerCase().includes(kw))
  );

  // Combine and deduplicate
  return [...new Map([...keywordResults, ...vectorResults].map(r => [r.id, r])).values()]
    .slice(0, limit);
}

3. Query Expansion

Ask the LLM to generate alternative queries:

async expandQuery(query: string): Promise<string[]> {
  const response = await this.openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{
      role: "user",
      content: `Generate 3 alternative ways to ask this question: "${query}". Return only the questions, one per line.`
    }],
  });

  return [query, ...response.choices[0].message.content.split('\n')];
}

Key Takeaways

1. RAG combines retrieval and generation for accurate, grounded answers
2. Document loading handles various formats and extracts text content
3. Chunking preserves context while creating searchable units
4. Vector stores enable semantic search to find relevant content
5. The QA chain orchestrates the full pipeline from question to answer
6. Source attribution builds trust by showing where answers come from

Resources

Congratulations!

You have built a complete RAG system from scratch. You now understand:

• How to load and process documents
• How to chunk text effectively
• How to store and search embeddings
• How to generate grounded answers with source attribution

In the next module, you will learn about AI Agents - systems that can reason, plan, and take actions autonomously.

Continue to Module 5: AI Agents