🚀 Distributed Streaming Platform

A production-grade distributed microservices streaming platform built using Spring Boot, Kafka, and Kubernetes.

This project demonstrates how modern backend systems are designed using:

• 🧩 Microservices Architecture
• ⚡ Event-Driven Communication (Kafka)
• 🔍 Service Discovery (Eureka)
• 🌐 API Gateway Routing
• ☸️ Kubernetes Deployment
• 🐳 Containerization with Docker

---

🎯 What This Project Solves

This system simulates a real-world video streaming backend where:

• Users upload content
• Content is processed asynchronously
• Streaming is handled efficiently
• Analytics are generated from events

---

💡 Why This Project Matters

This project is designed to reflect real industry-level system design, covering:

• Scalable microservices
• Distributed communication
• Fault tolerance challenges
• Kubernetes orchestration

---

🏗️ Architecture Overview

The system follows a microservices-based distributed architecture deployed on Kubernetes.

It is designed to handle high-throughput streaming workloads using asynchronous event-driven communication.

---

📊 High-Level System Architecture

---

🔍 Microservices Architecture

---

🧠 Core Architectural Concepts

1. API Gateway Pattern

All incoming requests are routed through a centralized API Gateway, which:

• Handles routing
• Provides a single entry point
• Simplifies client interaction

---

2. Service Discovery (Eureka)

Each microservice registers with the Discovery Server, enabling:

• Dynamic service lookup
• Loose coupling between services
• Resilience in distributed communication

---

3. Event-Driven Architecture (Kafka)

The system uses Apache Kafka for asynchronous communication:

• Content ingestion → produces events
• Streaming & analytics → consume events
• Decouples services for scalability

---

4. Database per Service Pattern

Each service manages its own data:

• MySQL → persistent storage
• Redis → caching layer
• MinIO → object storage (videos/files)

---

5. Kubernetes Orchestration

All components are deployed using Kubernetes:

• Deployments → stateless services
• StatefulSets → Kafka
• Services → internal networking
• Ingress → external routing

---

🔄 Request Flow (Simplified)

1. User hits streaming.local via Ingress
2. Request goes to API Gateway
3. Routed to respective microservice
4. Services communicate via:
   • REST (synchronous)
   • Kafka (asynchronous)
5. Data stored and processed across services

---

📁 Project Structure

distributed-streaming-platform/
│
├── api-gateway/              # API Gateway (Routing layer)
├── discovery-server/         # Eureka Service Registry
├── auth-service/             # Authentication & JWT
├── user-service/             # User management
├── content-service/          # Content metadata
├── content-ingestion-service/# Content upload pipeline
├── streaming-service/        # Streaming logic
├── video-processing-service/ # Video processing pipeline
├── analytic-service/         # Analytics & event processing
│
├── common-lib/               # Shared DTOs & utilities
├── kafka-kraft/              # Kafka setup (KRaft mode)
│
├── k8s/
│   ├── infrastructure/       # Kafka, MySQL, Redis, MinIO
│   ├── services/             # Microservice deployments
│   ├── ingress/              # Ingress configuration
│   └── namespace/            # Namespace setup
│
├── docker-compose.yaml       # Local development setup
├── init-db.sql               # Database initialization
├── images.txt                # List of Docker images
├── prometheus.yml           # Monitoring config
├── pom.xml                  # Parent Maven configuration
└── README.md

🧠 Structure Highlights

🔹 Modular Microservices

Each service is independently developed and deployed, enabling:

• Independent scaling
• Fault isolation
• Faster development cycles

---

🔹 Shared Library (common-lib)

Contains:

• DTOs
• Event classes
• Common utilities

👉 Ensures consistency across services

---

🔹 Infrastructure as Code (Kubernetes)

All infrastructure is defined declaratively under k8s/:

• Infrastructure layer (Kafka, DBs)
• Service deployments
• Networking (Ingress)

---

🔹 Hybrid Setup

• Docker Compose → Local development
• Kubernetes → Production-like deployment

---

⚙️ Tech Stack

🧠 Backend

• Java (Spring Boot) — Core microservices framework
• Spring Cloud — Service discovery & gateway
• Spring Security + JWT — Authentication & authorization

---

⚡ Messaging & Streaming

• Apache Kafka — Event-driven communication
• Kafka KRaft Mode — Zookeeper-less setup

---

🗄️ Databases & Storage

• MySQL — Relational database
• Redis — Caching layer
• MinIO — Object storage (video/files)

---

☸️ DevOps & Infrastructure

• Docker — Containerization
• Kubernetes (Minikube) — Container orchestration
• Ingress Controller (NGINX) — External routing

---

📊 Monitoring & Observability

• Prometheus — Metrics collection

---

🛠️ Build & Tools

• Maven — Build tool
• Docker Compose — Local setup

---

🧩 Microservices Overview

Service	Responsibility
API Gateway	Routes incoming requests
Discovery Server	Service registry (Eureka)
Auth Service	Authentication & JWT
User Service	User management
Content Service	Content metadata
Content Ingestion Service	Upload pipeline
Streaming Service	Streaming logic
Video Processing Service	Video processing
Analytics Service	Event analytics

---

📸 Screenshots

🟢 Kubernetes Pods (All Services Running)

!

---

🔌 Services (Internal Networking)

!

---

🚀 Key Highlights

• Built a distributed microservices system from scratch
• Implemented event-driven architecture using Kafka
• Deployed complete system on Kubernetes (Minikube)
• Integrated API Gateway + Service Discovery (Eureka)
• Designed for scalability, modularity, and fault isolation

---

🔮 Future Improvements

• Add CI/CD pipeline (GitHub Actions)
• Enable autoscaling (HPA)

---

🧑‍💻 Author

Yash chauhan
Backend Engineer | Distributed Systems Enthusiast

---

⭐ Support

If you found this project useful, consider giving it a ⭐ on GitHub.