Lesson 3.5: Practice - Async Data Loader

Duration: 60 minutes

Learning Objectives

By the end of this lesson, you will be able to:

1. Build a complete async data loader from scratch
2. Apply async/await patterns in a practical project
3. Implement error handling and retry logic
4. Create loading states and progress tracking
5. Type async operations correctly with TypeScript

Project Overview

In this practice lesson, you will build an AsyncDataLoader - a utility that fetches data from APIs with:

• Loading state management
• Error handling with retry logic
• Parallel and sequential loading options
• Caching support
• Progress callbacks
• Full TypeScript type safety

This combines everything you have learned in this module.

Step 1: Define the Types

First, let us define all the types we need:

// Possible states for a data loading operation
type LoadingState = 'idle' | 'loading' | 'success' | 'error';

// Result of a data fetch operation
interface DataResult<T> {
  data: T | null;
  error: Error | null;
  state: LoadingState;
  timestamp: number;
}

// Options for the data loader
interface LoaderOptions {
  retries?: number;
  retryDelay?: number;
  timeout?: number;
  cache?: boolean;
  cacheTtl?: number;
}

// Progress information
interface Progress {
  current: number;
  total: number;
  percentage: number;
}

// Callback types
type ProgressCallback = (progress: Progress) => void;
type StateChangeCallback<T> = (result: DataResult<T>) => void;

Step 2: Create Helper Functions

Build reusable utilities for our loader:

// Delay helper
function delay(ms: number): Promise<void> {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

// Timeout wrapper
async function withTimeout<T>(promise: Promise<T>, ms: number): Promise<T> {
  const timeoutPromise = new Promise<never>((_, reject) => {
    setTimeout(() => reject(new Error(`Operation timed out after ${ms}ms`)), ms);
  });

  return Promise.race([promise, timeoutPromise]);
}

// Create initial result
function createInitialResult<T>(): DataResult<T> {
  return {
    data: null,
    error: null,
    state: 'idle',
    timestamp: 0,
  };
}

// Create success result
function createSuccessResult<T>(data: T): DataResult<T> {
  return {
    data,
    error: null,
    state: 'success',
    timestamp: Date.now(),
  };
}

// Create error result
function createErrorResult<T>(error: Error): DataResult<T> {
  return {
    data: null,
    error,
    state: 'error',
    timestamp: Date.now(),
  };
}

Step 3: Build the Core Loader

Create the main AsyncDataLoader class:

class AsyncDataLoader<T> {
  private cache: Map<string, DataResult<T>> = new Map();
  private defaultOptions: Required<LoaderOptions> = {
    retries: 3,
    retryDelay: 1000,
    timeout: 10000,
    cache: true,
    cacheTtl: 60000, // 1 minute
  };

  constructor(private options: LoaderOptions = {}) {
    this.defaultOptions = { ...this.defaultOptions, ...options };
  }

  // Single item fetch with retries
  async fetch(
    key: string,
    fetcher: () => Promise<T>,
    options?: Partial<LoaderOptions>
  ): Promise<DataResult<T>> {
    const opts = { ...this.defaultOptions, ...options };

    // Check cache
    if (opts.cache) {
      const cached = this.cache.get(key);
      if (cached && this.isCacheValid(cached, opts.cacheTtl)) {
        return cached;
      }
    }

    // Attempt fetch with retries
    let lastError: Error | null = null;

    for (let attempt = 1; attempt <= opts.retries; attempt++) {
      try {
        const data = await withTimeout(fetcher(), opts.timeout);
        const result = createSuccessResult(data);

        if (opts.cache) {
          this.cache.set(key, result);
        }

        return result;
      } catch (error) {
        lastError = error instanceof Error ? error : new Error(String(error));

        if (attempt < opts.retries) {
          await delay(opts.retryDelay * attempt); // Exponential backoff
        }
      }
    }

    return createErrorResult(lastError!);
  }

  private isCacheValid(result: DataResult<T>, ttl: number): boolean {
    return result.state === 'success' && Date.now() - result.timestamp < ttl;
  }

  clearCache(): void {
    this.cache.clear();
  }

  invalidate(key: string): boolean {
    return this.cache.delete(key);
  }
}

Step 4: Add Batch Loading

Extend the loader to handle multiple items:

class AsyncDataLoader<T> {
  // ... previous code ...

  // Fetch multiple items in parallel
  async fetchMany(
    requests: Array<{ key: string; fetcher: () => Promise<T> }>,
    options?: Partial<LoaderOptions> & { onProgress?: ProgressCallback }
  ): Promise<Map<string, DataResult<T>>> {
    const results = new Map<string, DataResult<T>>();
    const total = requests.length;
    let completed = 0;

    const promises = requests.map(async ({ key, fetcher }) => {
      const result = await this.fetch(key, fetcher, options);
      results.set(key, result);

      completed++;
      if (options?.onProgress) {
        options.onProgress({
          current: completed,
          total,
          percentage: Math.round((completed / total) * 100),
        });
      }

      return result;
    });

    await Promise.all(promises);
    return results;
  }

  // Fetch multiple items sequentially
  async fetchSequential(
    requests: Array<{ key: string; fetcher: () => Promise<T> }>,
    options?: Partial<LoaderOptions> & {
      onProgress?: ProgressCallback;
      stopOnError?: boolean;
    }
  ): Promise<Map<string, DataResult<T>>> {
    const results = new Map<string, DataResult<T>>();
    const total = requests.length;

    for (let i = 0; i < requests.length; i++) {
      const { key, fetcher } = requests[i];
      const result = await this.fetch(key, fetcher, options);
      results.set(key, result);

      if (options?.onProgress) {
        options.onProgress({
          current: i + 1,
          total,
          percentage: Math.round(((i + 1) / total) * 100),
        });
      }

      if (options?.stopOnError && result.state === 'error') {
        break;
      }
    }

    return results;
  }
}

Step 5: Add Observable Pattern

Allow subscribing to state changes:

class AsyncDataLoader<T> {
  // ... previous code ...

  private subscribers: Map<string, Set<StateChangeCallback<T>>> = new Map();

  subscribe(key: string, callback: StateChangeCallback<T>): () => void {
    if (!this.subscribers.has(key)) {
      this.subscribers.set(key, new Set());
    }

    this.subscribers.get(key)!.add(callback);

    // Return unsubscribe function
    return () => {
      this.subscribers.get(key)?.delete(callback);
    };
  }

  private notify(key: string, result: DataResult<T>): void {
    const callbacks = this.subscribers.get(key);
    if (callbacks) {
      callbacks.forEach((callback) => callback(result));
    }
  }

  // Updated fetch method with notifications
  async fetchWithNotify(
    key: string,
    fetcher: () => Promise<T>,
    options?: Partial<LoaderOptions>
  ): Promise<DataResult<T>> {
    // Notify loading state
    this.notify(key, {
      data: null,
      error: null,
      state: 'loading',
      timestamp: Date.now(),
    });

    const result = await this.fetch(key, fetcher, options);

    // Notify final state
    this.notify(key, result);

    return result;
  }
}

Step 6: Complete Implementation

Here is the full implementation with all features:

// Types
type LoadingState = 'idle' | 'loading' | 'success' | 'error';

interface DataResult<T> {
  data: T | null;
  error: Error | null;
  state: LoadingState;
  timestamp: number;
}

interface LoaderOptions {
  retries?: number;
  retryDelay?: number;
  timeout?: number;
  cache?: boolean;
  cacheTtl?: number;
}

interface Progress {
  current: number;
  total: number;
  percentage: number;
}

type ProgressCallback = (progress: Progress) => void;
type StateChangeCallback<T> = (result: DataResult<T>) => void;

// Helpers
function delay(ms: number): Promise<void> {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

async function withTimeout<T>(promise: Promise<T>, ms: number): Promise<T> {
  const timeoutPromise = new Promise<never>((_, reject) => {
    setTimeout(() => reject(new Error(`Timeout after ${ms}ms`)), ms);
  });
  return Promise.race([promise, timeoutPromise]);
}

// Main class
class AsyncDataLoader<T> {
  private cache = new Map<string, DataResult<T>>();
  private subscribers = new Map<string, Set<StateChangeCallback<T>>>();
  private options: Required<LoaderOptions>;

  constructor(options: LoaderOptions = {}) {
    this.options = {
      retries: 3,
      retryDelay: 1000,
      timeout: 10000,
      cache: true,
      cacheTtl: 60000,
      ...options,
    };
  }

  async fetch(
    key: string,
    fetcher: () => Promise<T>,
    options?: Partial<LoaderOptions>
  ): Promise<DataResult<T>> {
    const opts = { ...this.options, ...options };

    // Check cache
    if (opts.cache) {
      const cached = this.cache.get(key);
      if (cached && cached.state === 'success') {
        const age = Date.now() - cached.timestamp;
        if (age < opts.cacheTtl) {
          return cached;
        }
      }
    }

    // Notify loading
    this.notify(key, { data: null, error: null, state: 'loading', timestamp: Date.now() });

    // Attempt with retries
    let lastError: Error | null = null;

    for (let attempt = 1; attempt <= opts.retries; attempt++) {
      try {
        const data = await withTimeout(fetcher(), opts.timeout);
        const result: DataResult<T> = {
          data,
          error: null,
          state: 'success',
          timestamp: Date.now(),
        };

        if (opts.cache) {
          this.cache.set(key, result);
        }

        this.notify(key, result);
        return result;
      } catch (error) {
        lastError = error instanceof Error ? error : new Error(String(error));

        if (attempt < opts.retries) {
          await delay(opts.retryDelay * attempt);
        }
      }
    }

    const errorResult: DataResult<T> = {
      data: null,
      error: lastError!,
      state: 'error',
      timestamp: Date.now(),
    };

    this.notify(key, errorResult);
    return errorResult;
  }

  async fetchMany(
    requests: Array<{ key: string; fetcher: () => Promise<T> }>,
    options?: Partial<LoaderOptions> & { onProgress?: ProgressCallback }
  ): Promise<Map<string, DataResult<T>>> {
    const results = new Map<string, DataResult<T>>();
    let completed = 0;
    const total = requests.length;

    await Promise.all(
      requests.map(async ({ key, fetcher }) => {
        const result = await this.fetch(key, fetcher, options);
        results.set(key, result);
        completed++;
        options?.onProgress?.({
          current: completed,
          total,
          percentage: Math.round((completed / total) * 100),
        });
      })
    );

    return results;
  }

  subscribe(key: string, callback: StateChangeCallback<T>): () => void {
    if (!this.subscribers.has(key)) {
      this.subscribers.set(key, new Set());
    }
    this.subscribers.get(key)!.add(callback);
    return () => this.subscribers.get(key)?.delete(callback);
  }

  private notify(key: string, result: DataResult<T>): void {
    this.subscribers.get(key)?.forEach((cb) => cb(result));
  }

  clearCache(): void {
    this.cache.clear();
  }

  invalidate(key: string): boolean {
    return this.cache.delete(key);
  }
}

Step 7: Usage Examples

Basic Usage

interface User {
  id: number;
  name: string;
  email: string;
}

const userLoader = new AsyncDataLoader<User>({
  retries: 3,
  timeout: 5000,
  cacheTtl: 300000, // 5 minutes
});

// Fetch a single user
async function getUser(id: number): Promise<User | null> {
  const result = await userLoader.fetch(`user:${id}`, async () => {
    const response = await fetch(`https://jsonplaceholder.typicode.com/users/${id}`);
    if (!response.ok) throw new Error(`HTTP ${response.status}`);
    return response.json();
  });

  if (result.state === 'error') {
    console.error('Failed to fetch user:', result.error?.message);
    return null;
  }

  return result.data;
}

Batch Loading with Progress

async function loadAllUsers(): Promise<void> {
  const userIds = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];

  const results = await userLoader.fetchMany(
    userIds.map((id) => ({
      key: `user:${id}`,
      fetcher: async () => {
        const response = await fetch(`https://jsonplaceholder.typicode.com/users/${id}`);
        return response.json();
      },
    })),
    {
      onProgress: (progress) => {
        console.log(
          `Loading users: ${progress.percentage}% (${progress.current}/${progress.total})`
        );
      },
    }
  );

  // Process results
  let successful = 0;
  let failed = 0;

  results.forEach((result, key) => {
    if (result.state === 'success') {
      successful++;
      console.log(`Loaded ${key}:`, result.data?.name);
    } else {
      failed++;
      console.error(`Failed ${key}:`, result.error?.message);
    }
  });

  console.log(`Complete: ${successful} loaded, ${failed} failed`);
}

With State Subscriptions

// Subscribe to user updates
const unsubscribe = userLoader.subscribe('user:1', (result) => {
  switch (result.state) {
    case 'loading':
      console.log('Loading user...');
      break;
    case 'success':
      console.log('User loaded:', result.data?.name);
      break;
    case 'error':
      console.error('Error:', result.error?.message);
      break;
  }
});

// Fetch will trigger notifications
await userLoader.fetch('user:1', fetchUser1);

// Cleanup when done
unsubscribe();

Challenge: Extend the Loader

Try adding these features to your implementation:

1. Debouncing: Prevent duplicate requests for the same key within a short window
2. Request deduplication: If the same key is being fetched, return the existing Promise
3. Background refresh: Refresh stale cache entries in the background
4. Batch API support: Combine multiple requests into a single API call

Example: Request Deduplication

class AsyncDataLoader<T> {
  private inflight = new Map<string, Promise<DataResult<T>>>();

  async fetch(key: string, fetcher: () => Promise<T>): Promise<DataResult<T>> {
    // Return existing request if in flight
    const existing = this.inflight.get(key);
    if (existing) {
      return existing;
    }

    // Create new request
    const promise = this.doFetch(key, fetcher);
    this.inflight.set(key, promise);

    try {
      return await promise;
    } finally {
      this.inflight.delete(key);
    }
  }

  private async doFetch(key: string, fetcher: () => Promise<T>): Promise<DataResult<T>> {
    // Original fetch logic
  }
}

Key Takeaways

1. Modular design: Break complex async logic into small, testable functions
2. Type everything: Use TypeScript interfaces to define all data structures
3. Handle all states: Account for loading, success, and error states
4. Provide options: Make retry count, timeout, and caching configurable
5. Progress feedback: Provide callbacks for long-running operations
6. Observable pattern: Allow subscribers to react to state changes
7. Caching: Avoid redundant network requests with smart caching

Resources

Module Complete

Congratulations! You have completed Module 3: Async/Await. You now understand:

• How to write async functions with async/await syntax
• Error handling with try/catch in async code
• Running operations in parallel with Promise.all and Promise.allSettled
• Properly typing async functions in TypeScript
• Building practical async utilities

Continue to Module 4: HTTP and REST API to learn how to work with real APIs.