TaskFlow — Distributed Job Processing Platform

TaskFlow is a robust, production-ready distributed job processing platform. It is designed to handle asynchronous background tasks with high reliability, featuring a real-time monitoring dashboard, advanced concurrency control, and automatic retry mechanisms.

✨ Features

• Distributed Workers: Easily scale out background processing with decoupled Node.js workers using BullMQ.
• State Management: PostgreSQL serves as the absolute source of truth for job states (QUEUED, PROCESSING, COMPLETED, DEAD), idempotency, and lock management.
• Real-Time Dashboard: A vanilla JavaScript dashboard powered by Server-Sent Events (SSE) provides live insights into job statuses, worker health, and system metrics.
• Resiliency & Retries: Automatic exponential backoff retries for failed jobs, graceful shutdowns handling SIGTERM/SIGINT, and manual replay capability for DEAD jobs.
• Production-Ready Tooling:
  • Input validation powered by Zod.
  • High-performance structured JSON logging with Pino.
  • Automated CI pipeline configured via GitHub Actions.

🏗 System Architecture

flowchart TB
    Client[Client / Dashboard]
    
    subgraph Application Tier
        API[Express REST API]
        Worker[Background Worker]
    end
    
    subgraph Storage Tier
        PG[(PostgreSQL)]
        Redis[(Redis / BullMQ)]
    end

    Client <-->|"HTTP POST / SSE"| API
    
    API -->|"Validates & Saves State"| PG
    API -->|"Enqueues Job"| Redis
    
    Worker -->|"Listens for Jobs"| Redis
    Worker -->|"Claims & Updates State"| PG

Loading

🚀 Quick Start

Prerequisites

• Docker & Docker Compose
• Node.js (v20+ recommended)

1. Spin up Infrastructure

Start PostgreSQL and Redis in the background using Docker Compose:

docker-compose up -d postgres redis

2. Start the Backend API

cd backend
npm install
npm start

The REST API and SSE event stream will run on http://localhost:5000.

3. Start a Worker

Open a new terminal window to spin up a worker instance (you can run multiple to test concurrency):

cd worker
npm install
npm start

4. View the Dashboard

Simply open frontend/index.html in your web browser. The dashboard will automatically connect to the backend and stream live metrics.

🛠 Tech Stack

• Backend: Node.js, Express.js
• Queue: Redis, BullMQ
• Database: PostgreSQL (pg)
• Frontend: HTML5, Vanilla JavaScript, CSS3, Server-Sent Events (SSE)
• Validation & Logging: Zod, Pino

🧪 Testing

The project includes an extensive test suite verifying database concurrency, queue recovery, and job states.

cd backend
npm test

Note: Ensure Docker containers are running before executing tests.

🔒 Security Note

This project is currently optimized for demonstration and internal network usage. For public production deployment, ensure you configure proper JWT authentication for the dashboard and tighten CORS policies in the Express configuration.