Lesson 2.5: Hallucinations and Their Causes

Duration: 50 minutes

Learning Objectives

By the end of this lesson, you will be able to:

• Define what hallucinations are in the context of LLMs
• Understand why LLMs hallucinate
• Identify different types of hallucinations
• Implement strategies to detect hallucinations
• Apply techniques to reduce hallucinations in your applications

Introduction

LLMs can be incredibly helpful, but they have a serious flaw: they can confidently generate information that is completely false. This is called "hallucination" - the model produces plausible-sounding but incorrect or fabricated content.

Understanding hallucinations is critical for building reliable AI applications. In this lesson, we will explore why hallucinations happen and what you can do about them.

What is a Hallucination?

A hallucination occurs when an LLM generates content that is:

• Factually incorrect
• Fabricated (made up entirely)
• Inconsistent with provided context
• Logically impossible

Examples of Hallucinations

┌─────────────────────────────────────────────────────────┐
│                 Hallucination Examples                   │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  User: "Who wrote the book 'The Azure Chronicles'?"     │
│                                                         │
│  LLM: "The Azure Chronicles was written by             │
│        Jonathan Maxwell in 2019. It's a fantasy        │
│        novel about a kingdom in the clouds."           │
│                                                         │
│  Reality: This book does not exist. The model          │
│           fabricated the title, author, and plot.      │
│                                                         │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  User: "What's the population of Springfield?"          │
│                                                         │
│  LLM: "Springfield has a population of 167,882         │
│        as of the 2020 census."                         │
│                                                         │
│  Reality: There are many Springfields. The model       │
│           gave a confident but arbitrary answer.       │
│                                                         │
└─────────────────────────────────────────────────────────┘

Why "Hallucination"?

The term comes from the analogy to human hallucinations - perceiving things that are not there. LLMs "perceive" patterns in their training data and extrapolate, sometimes creating things that do not exist.

Why LLMs Hallucinate

Understanding the causes helps you prevent and detect hallucinations.

Cause 1: Training Data Patterns

LLMs learn statistical patterns, not facts. When asked about something unfamiliar, they extrapolate from similar patterns:

Training data contained many sentences like:
  "The novel [Title] was written by [Author] in [Year]"

When asked about a non-existent book:
  Model fills in the pattern with plausible values
  It does not "know" whether the book exists
  It generates what statistically "should" come next

Cause 2: No Concept of Truth

LLMs have no internal fact database. They cannot verify claims against reality:

// What the LLM does NOT do:
function answer(question: string): string {
  const fact = factDatabase.lookup(question); // No such database
  if (fact.verified) {
    return fact.value;
  }
}

// What the LLM actually does:
function answer(question: string): string {
  return generateMostProbableTokenSequence(question);
  // Probability is based on training data patterns
  // Not on factual accuracy
}

Cause 3: Confidence Without Certainty

LLMs always produce confident-sounding output because:

• They are trained to generate fluent, natural text
• Hedging language ("I'm not sure") is statistically less common
• They cannot assess their own knowledge gaps

Human: "What year did Napoleon visit America?"

LLM: "Napoleon visited America in 1803 during..."

Reality: Napoleon never visited America.
         The model confidently invented a date.

Cause 4: Training Data Errors

LLMs learn from internet text, which contains errors:

• Outdated information
• Biased or incorrect sources
• Fictional content mixed with facts
• Jokes and sarcasm taken literally

Cause 5: Compression and Generalization

The model compresses vast knowledge into parameters:

• Details get blurred
• Similar concepts get merged
• Specific facts become approximations

Training: Thousands of articles about various scientists
Result: Model learns "scientist patterns"
Error: May attribute discoveries to the wrong person
       (Einstein, Newton, etc. get mixed up on edge cases)

Types of Hallucinations

Type 1: Factual Hallucinations

Making up or getting facts wrong:

User: "What is the capital of Australia?"
LLM: "The capital of Australia is Sydney."
Correct: The capital is Canberra.

Type 2: Fabrication

Inventing entities that do not exist:

User: "Tell me about the TypeScript method array.sortBy()"
LLM: "The sortBy() method sorts array elements by a
      specified property. It was added in TypeScript 4.5..."
Reality: This method does not exist in TypeScript/JavaScript.

Type 3: Inconsistency

Contradicting itself or the provided context:

User: "My name is Alice."
LLM: "Nice to meet you, Alice!"
User: "What's my name?"
LLM: "Your name is Bob."

Type 4: Citation Hallucinations

Inventing fake references:

User: "Cite a source for the claim that TypeScript
       improves code quality."
LLM: "According to Smith et al. (2022) in their paper
      'TypeScript Adoption in Enterprise' published in
      IEEE Software..."
Reality: This paper does not exist.

Type 5: Logical Hallucinations

Generating logically impossible content:

User: "If A is greater than B, and B is greater than C,
       what can we say about A and C?"
LLM: "C is greater than A."
Correct: A is greater than C.

Detecting Hallucinations

Strategy 1: Self-Consistency Checking

Ask the model the same question multiple times with temperature > 0:

async function checkConsistency(
  question: string,
  numChecks: number = 3
): Promise<{
  responses: string[];
  consistent: boolean;
}> {
  const responses: string[] = [];

  for (let i = 0; i < numChecks; i++) {
    const response = await openai.chat.completions.create({
      model: 'gpt-4o-mini',
      temperature: 0.7, // Allow variation
      messages: [{ role: 'user', content: question }],
    });
    responses.push(response.choices[0].message.content || '');
  }

  // Check if responses agree on key facts
  const consistent = checkResponseAgreement(responses);

  return { responses, consistent };
}

function checkResponseAgreement(responses: string[]): boolean {
  // Simple check: look for contradictions
  // In production, you would use more sophisticated comparison
  const firstResponse = responses[0].toLowerCase();

  return responses.every((r) => {
    // Check if key numbers/names match
    const numbers1 = firstResponse.match(/\d+/g) || [];
    const numbers2 = r.toLowerCase().match(/\d+/g) || [];
    return JSON.stringify(numbers1) === JSON.stringify(numbers2);
  });
}

Strategy 2: Ask for Sources

Request citations and verify them:

const response = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [
    {
      role: 'system',
      content: `When making factual claims, cite specific sources.
                Format: [Source: URL or "Author, Title, Year"]
                If you cannot cite a source, say "I cannot verify this."`,
    },
    {
      role: 'user',
      content: 'What percentage of JavaScript developers use TypeScript?',
    },
  ],
});

// Then verify the sources actually exist
// Note: This does not guarantee the source says what the model claims!

Strategy 3: Structured Output Verification

Use structured output and validate:

interface FactualClaim {
  claim: string;
  confidence: 'high' | 'medium' | 'low';
  source?: string;
  needsVerification: boolean;
}

const response = await openai.chat.completions.create({
  model: 'gpt-4o',
  response_format: { type: 'json_object' },
  messages: [
    {
      role: 'system',
      content: `Respond with JSON containing factual claims.
                For each claim, assess your confidence.
                Mark claims that need human verification.
                Example: {
                  "claims": [
                    {
                      "claim": "TypeScript was released in 2012",
                      "confidence": "high",
                      "source": "Microsoft official announcement",
                      "needsVerification": false
                    }
                  ]
                }`,
    },
    { role: 'user', content: "Tell me about TypeScript's history" },
  ],
});

const data = JSON.parse(response.choices[0].message.content || '{}');

// Flag low-confidence claims for review
const needsReview = data.claims.filter(
  (c: FactualClaim) => c.confidence === 'low' || c.needsVerification
);

Strategy 4: Knowledge Cutoff Awareness

Remember that LLMs have training cutoff dates:

const KNOWLEDGE_CUTOFF = new Date('2024-01-01'); // Example

function mightBeOutdated(question: string): boolean {
  const currentEventIndicators = [
    'latest',
    'newest',
    'current',
    'today',
    'this year',
    '2024',
    '2025',
    'recently',
  ];

  const questionLower = question.toLowerCase();
  return currentEventIndicators.some((indicator) => questionLower.includes(indicator));
}

async function answerWithCutoffWarning(question: string): Promise<string> {
  const response = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: [{ role: 'user', content: question }],
  });

  let answer = response.choices[0].message.content || '';

  if (mightBeOutdated(question)) {
    answer +=
      '\n\n⚠️ Note: This information may be outdated. ' +
      'My knowledge was last updated in early 2024.';
  }

  return answer;
}

Reducing Hallucinations

Technique 1: Provide Context (RAG)

Give the model the facts it needs:

// BAD: Asking about something not in training data
const badResponse = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: "What are your company's refund policies?" }],
});
// Model will hallucinate a policy!

// GOOD: Provide the actual policy
const goodResponse = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [
    {
      role: 'system',
      content: `Answer based ONLY on the provided policy document.
                If the answer is not in the document, say "I don't have 
                information about that in the policy."`,
    },
    {
      role: 'user',
      content: `Policy Document:
                ${actualPolicyDocument}
                
                Question: What are your company's refund policies?`,
    },
  ],
});

Technique 2: Use Lower Temperature

For factual tasks, reduce randomness:

// High temperature = more creative = more hallucination risk
const riskyResponse = await openai.chat.completions.create({
  model: 'gpt-4o',
  temperature: 1.0,
  messages: [{ role: 'user', content: 'What year was TypeScript released?' }],
});

// Low temperature = more deterministic = less hallucination risk
const saferResponse = await openai.chat.completions.create({
  model: 'gpt-4o',
  temperature: 0,
  messages: [{ role: 'user', content: 'What year was TypeScript released?' }],
});

Technique 3: Explicit Uncertainty Instructions

Tell the model it is okay to say "I don't know":

const response = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [
    {
      role: 'system',
      content: `You are a helpful assistant. Important guidelines:
                - If you're not sure about something, say so
                - If a question is ambiguous, ask for clarification
                - Never make up facts or citations
                - It's better to say "I don't know" than to guess`,
    },
    { role: 'user', content: 'Who won the 2025 Super Bowl?' },
  ],
});
// Model should indicate it does not have this information

Technique 4: Chain of Thought Prompting

Ask the model to show its reasoning:

const response = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [
    {
      role: 'system',
      content: `Before answering, think through the question step by step.
                Show your reasoning. If any step is uncertain, note that.`,
    },
    {
      role: 'user',
      content: 'Is TypeScript a superset of JavaScript?',
    },
  ],
});

// Response will include reasoning, making errors more visible:
// "Let me think through this:
//  1. TypeScript was created by Microsoft
//  2. TypeScript files can contain valid JavaScript
//  3. TypeScript adds type annotations to JavaScript
//  4. Therefore, TypeScript is a superset of JavaScript
//  Answer: Yes, TypeScript is a superset of JavaScript."

Technique 5: Grounding with Tools

Use function calling to access real data:

const tools = [
  {
    type: 'function' as const,
    function: {
      name: 'get_current_weather',
      description: 'Get the current weather in a location',
      parameters: {
        type: 'object',
        properties: {
          location: { type: 'string', description: 'City name' },
        },
        required: ['location'],
      },
    },
  },
];

// Model can call real APIs instead of hallucinating data
const response = await openai.chat.completions.create({
  model: 'gpt-4o',
  tools,
  messages: [{ role: 'user', content: "What's the weather in Tokyo?" }],
});

// If model calls the tool, you get real data
// If it tries to answer directly, you know to be skeptical

Building Hallucination-Resistant Applications

Pattern: Verify Before Display

interface VerifiedResponse {
  content: string;
  verified: boolean;
  warnings: string[];
}

async function getVerifiedResponse(question: string): Promise<VerifiedResponse> {
  const warnings: string[] = [];

  // Get initial response
  const response = await openai.chat.completions.create({
    model: 'gpt-4o',
    temperature: 0,
    messages: [
      {
        role: 'system',
        content: 'Answer factually. Express uncertainty when appropriate.',
      },
      { role: 'user', content: question },
    ],
  });

  const content = response.choices[0].message.content || '';

  // Check for uncertainty markers
  const uncertaintyPhrases = [
    "I'm not sure",
    'I believe',
    'might be',
    'possibly',
    'I think',
    'approximately',
  ];

  const hasUncertainty = uncertaintyPhrases.some((phrase) =>
    content.toLowerCase().includes(phrase)
  );

  if (hasUncertainty) {
    warnings.push('Response contains uncertain claims');
  }

  // Check for specific numbers or dates that need verification
  const hasSpecificNumbers = /\b\d{4,}\b/.test(content);
  if (hasSpecificNumbers) {
    warnings.push('Contains specific numbers that should be verified');
  }

  return {
    content,
    verified: warnings.length === 0,
    warnings,
  };
}

Pattern: Human-in-the-Loop

interface ReviewableResponse {
  response: string;
  needsReview: boolean;
  reviewReasons: string[];
}

async function generateWithReviewFlag(
  prompt: string,
  highStakesTopics: string[]
): Promise<ReviewableResponse> {
  const response = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: [{ role: 'user', content: prompt }],
  });

  const content = response.choices[0].message.content || '';
  const reviewReasons: string[] = [];

  // Check if response touches high-stakes topics
  for (const topic of highStakesTopics) {
    if (content.toLowerCase().includes(topic.toLowerCase())) {
      reviewReasons.push(`Contains high-stakes topic: ${topic}`);
    }
  }

  // Check for medical/legal/financial advice patterns
  const sensitivePatterns = [
    /you should (take|stop taking) medication/i,
    /legal advice/i,
    /invest in/i,
    /guaranteed returns/i,
  ];

  for (const pattern of sensitivePatterns) {
    if (pattern.test(content)) {
      reviewReasons.push('Contains potentially sensitive advice');
      break;
    }
  }

  return {
    response: content,
    needsReview: reviewReasons.length > 0,
    reviewReasons,
  };
}

// Usage
const result = await generateWithReviewFlag('What should I do about my headache?', [
  'medication',
  'diagnosis',
  'treatment',
]);

if (result.needsReview) {
  console.log('⚠️ This response needs human review:');
  console.log(result.reviewReasons);
}

Exercises

Exercise 1: Identify Hallucination Types

Classify each hallucination:

1. LLM says "JavaScript was created in 1995 by Brendan Eich at Mozilla"
2. LLM invents a npm package called "super-validator" that does not exist
3. LLM says a user's name is "John" when they said it was "Jane"
4. LLM cites "The TypeScript Handbook, 3rd Edition, O'Reilly, 2023"

Exercise 2: Write Anti-Hallucination Prompts

Rewrite these prompts to reduce hallucination risk:

1. "Tell me about recent advances in quantum computing"
2. "What's the best TypeScript library for validation?"

Explain quantum computing concepts you are confident about.
Note: My knowledge has a cutoff date, so I may not know about the
most recent developments. For the latest advances, please verify
with current sources like arXiv or major tech company blogs.

What foundational concepts in quantum computing can you explain?

List well-known TypeScript validation libraries. For each:
- Name and what it does
- Only mention libraries you are confident exist
- If you're unsure about a library, indicate that
- Do not make up download statistics or exact version numbers

I will verify these libraries exist on npm before using them.

Exercise 3: Build a Hallucination Detector

Implement a simple function that checks if a response might contain hallucinations:

function detectPotentialHallucination(response: string): {
  suspicious: boolean;
  reasons: string[];
} {
  // Your implementation
}

function detectPotentialHallucination(response: string): {
  suspicious: boolean;
  reasons: string[];
} {
  const reasons: string[] = [];
  const responseLower = response.toLowerCase();

  // Check for overly specific numbers (often made up)
  const specificNumbers = response.match(/\b\d{5,}\b/g);
  if (specificNumbers && specificNumbers.length > 0) {
    reasons.push(`Contains very specific numbers: ${specificNumbers.join(', ')}`);
  }

  // Check for fake-sounding citations
  const citationPattern = /according to .+ \(\d{4}\)|.+ et al\./i;
  if (citationPattern.test(response)) {
    reasons.push('Contains citation that should be verified');
  }

  // Check for definitive claims about uncertain topics
  const definitivePatterns = [
    /the best .+ is definitely/i,
    /always use/i,
    /never use/i,
    /100% of/i,
    /everyone agrees/i,
  ];

  for (const pattern of definitivePatterns) {
    if (pattern.test(response)) {
      reasons.push('Contains overly definitive claims');
      break;
    }
  }

  // Check for future predictions presented as fact
  const futureYears = response.match(/\b(2025|2026|2027|2028|2029|2030)\b/g);
  if (
    futureYears &&
    !responseLower.includes('might') &&
    !responseLower.includes('could') &&
    !responseLower.includes('prediction')
  ) {
    reasons.push('Contains future dates stated as fact');
  }

  return {
    suspicious: reasons.length > 0,
    reasons,
  };
}

// Test
const result = detectPotentialHallucination(
  'According to Smith et al. (2023), TypeScript will have 50000000 ' +
    'users by 2026. The best validation library is definitely Zod.'
);

console.log(result);
// { suspicious: true, reasons: [
//   "Contains very specific numbers: 50000000",
//   "Contains citation that should be verified",
//   "Contains overly definitive claims",
//   "Contains future dates stated as fact"
// ]}

Key Takeaways

1. Hallucinations are inevitable - LLMs predict probable text, not truth
2. LLMs cannot verify facts - they have no concept of truth, only patterns
3. Confidence does not equal accuracy - models always sound confident
4. Different types exist: factual errors, fabrications, inconsistencies, fake citations
5. Detection strategies: consistency checking, requesting sources, structured validation
6. Prevention techniques: provide context (RAG), lower temperature, encourage uncertainty, use tools
7. Always verify critical information from external sources
8. Design for failure - assume hallucinations will happen and build safeguards

Resources

Module Summary

Congratulations on completing Module 2! You now understand:

• What LLMs are and how they generate text
• How tokens work and affect cost
• Context window limitations and management strategies
• How to tune temperature and other parameters
• Why hallucinations occur and how to mitigate them

This knowledge prepares you to work effectively with AI APIs. In the next module, you will learn prompt engineering - the art of getting LLMs to do exactly what you want.

Continue to Module 3: Prompt Engineering