Event-Driven Spring Boot Microservices with Apache Kafka

This project demonstrates a complete microservices architecture using Apache Kafka for event-driven communication, featuring three specialized microservices with different build systems and databases, plus an Angular frontend.

Architecture Overview

Microservices

Accounts Service (Maven + PostgreSQL) - User account management
TextAdventure Service (Gradle + MongoDB) - Game adventures and questions
Gameplays Service (Pure Java + MySQL) - Gameplay session storage
Frontend (Angular) - Web interface for all services

Infrastructure

Apache Kafka Cluster - 3 brokers for event streaming
PostgreSQL - Accounts database
MongoDB - TextAdventures database
MySQL - Gameplays database
Nginx - API Gateway and load balancer
Vagrant - Multi-VM production simulation

Deployment Options

Option 1: Development Setup (Docker Compose)

Simple setup for development and testing:

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Accounts      │    │  TextAdventures │    │   Gameplays     │
│   Service       │    │    Service      │    │    Service      │
│   (Maven)       │    │   (Gradle)      │    │  (Pure Java)    │
│   PostgreSQL    │    │   MongoDB       │    │    MySQL        │
│   Port: 8081    │    │   Port: 8082    │    │   Port: 8083    │
└─────────┬───────┘    └─────────┬───────┘    └─────────┬───────┘
          │                      │                      │
          └──────────────────────┼──────────────────────┘
                                 │
                    ┌─────────────────┐
                    │  Kafka Cluster  │
                    │  (3 Brokers)    │
                    │  9092,9094,9096 │
                    └─────────────────┘
                                 │
                    ┌─────────────────┐
                    │  Angular        │
                    │  Frontend       │
                    │  Port: 4200     │
                    └─────────────────┘

Option 2: Production Simulation (Vagrant + Nginx)

Multi-VM setup simulating real production environment:

┌─────────────────────────────────────────────────────────────┐
│                        External Access                      │
│                     https://api.local                       │
└─────────────────────┬───────────────────────────────────────┘
                      │
         ┌─────────────────┐
         │      VM-1       │
         │  Nginx Gateway  │ :80/:443
         │   + Frontend    │ (Angular)
         │ 192.168.50.11   │
         └─────────┬───────┘
                   │
    ┌──────────────┼──────────────┐
    │              │              │
┌───▼────┐    ┌───▼────┐    ┌───▼────┐
│  VM-2  │    │  VM-3  │    │  VM-4  │
│Accounts│    │TextAdv │    │Gameplay│
│+PostSQL│    │+MongoDB│    │+MySQL  │
│:8081   │    │:8082   │    │:8083   │
│.50.12  │    │.50.13  │    │.50.14  │
└────────┘    └────────┘    └────────┘
                   │
         ┌─────────▼───────┐
         │      VM-5       │
         │ Kafka Cluster   │
         │ (3 Brokers)     │
         │ 192.168.50.15   │
         └─────────────────┘

Quick Start

Prerequisites

For Development Setup (Option 1)

Docker & Docker Compose
Java 24
Node.js & npm
Maven
Gradle

For Production Simulation (Option 2)

All above prerequisites
Vagrant
VirtualBox
8GB+ RAM recommended
20GB+ free disk space

Setup Instructions

Option 1: Development Setup

1. Start Infrastructure

# Start Kafka cluster and databases
docker-compose up -d

# Verify Kafka cluster
docker-compose ps

2. Start Microservices

Accounts Service (Maven)

cd accounts-service
mvn spring-boot:run

TextAdventure Service (Gradle)

cd textadventure-service
./gradlew bootRun

Gameplays Service (Pure Java)

cd gameplays-service
javac -cp "lib/*" src/main/java/com/microservices/gameplays/*.java
java -cp "lib/*:src/main/java" com.microservices.gameplays.GameplaysApplication

Frontend (Angular)

cd frontend
npm install
ng serve

Option 2: Production Simulation (Vagrant + Nginx)

1. Create Vagrant Configuration

Create Vagrantfile in project root:

Vagrant.configure("2") do |config|
  # VM-1: Nginx Gateway + Frontend
  config.vm.define "gateway" do |gateway|
    gateway.vm.box = "ubuntu/20.04"
    gateway.vm.network "private_network", ip: "192.168.50.11"
    gateway.vm.provider "virtualbox" do |vb|
      vb.memory = "1024"
      vb.cpus = 2
    end
    gateway.vm.provision "shell", inline: <<-SHELL
      apt-get update
      apt-get install -y nginx nodejs npm openjdk-24-jdk
      # Configure Nginx as reverse proxy
    SHELL
  end

  # VM-2: Accounts Service + PostgreSQL
  config.vm.define "accounts" do |accounts|
    accounts.vm.box = "ubuntu/20.04"
    accounts.vm.network "private_network", ip: "192.168.50.12"
    accounts.vm.provider "virtualbox" do |vb|
      vb.memory = "2048"
      vb.cpus = 2
    end
    accounts.vm.provision "shell", inline: <<-SHELL
      apt-get update
      apt-get install -y openjdk-24-jdk maven postgresql postgresql-contrib
      # Configure PostgreSQL and deploy service
    SHELL
  end

  # VM-3: TextAdventure Service + MongoDB
  config.vm.define "textadv" do |textadv|
    textadv.vm.box = "ubuntu/20.04"
    textadv.vm.network "private_network", ip: "192.168.50.13"
    textadv.vm.provider "virtualbox" do |vb|
      vb.memory = "2048"
      vb.cpus = 2
    end
    textadv.vm.provision "shell", inline: <<-SHELL
      apt-get update
      apt-get install -y openjdk-24-jdk gradle mongodb
      # Configure MongoDB and deploy service
    SHELL
  end

  # VM-4: Gameplays Service + MySQL
  config.vm.define "gameplays" do |gameplays|
    gameplays.vm.box = "ubuntu/20.04"
    gameplays.vm.network "private_network", ip: "192.168.50.14"
    gameplays.vm.provider "virtualbox" do |vb|
      vb.memory = "2048"
      vb.cpus = 2
    end
    gameplays.vm.provision "shell", inline: <<-SHELL
      apt-get update
      apt-get install -y openjdk-24-jdk mysql-server
      # Configure MySQL and deploy service
    SHELL
  end

  # VM-5: Kafka Cluster
  config.vm.define "kafka" do |kafka|
    kafka.vm.box = "ubuntu/20.04"
    kafka.vm.network "private_network", ip: "192.168.50.15"
    kafka.vm.provider "virtualbox" do |vb|
      vb.memory = "3072"
      vb.cpus = 3
    end
    kafka.vm.provision "shell", inline: <<-SHELL
      apt-get update
      apt-get install -y openjdk-24-jdk docker.io docker-compose
      # Deploy Kafka cluster via Docker
    SHELL
  end
end

2. Create Nginx Configuration

Create nginx/nginx.conf:

upstream accounts_backend {
    server 192.168.50.12:8081;
}

upstream textadv_backend {
    server 192.168.50.13:8082;
}

upstream gameplays_backend {
    server 192.168.50.14:8083;
}

server {
    listen 80;
    server_name api.local;

    # API Gateway routes
    location /api/accounts/ {
        proxy_pass http://accounts_backend/api/accounts/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    location /api/adventures/ {
        proxy_pass http://textadv_backend/api/adventures/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    location /api/gameplays/ {
        proxy_pass http://gameplays_backend/api/gameplays/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    # Frontend
    location / {
        root /var/www/html;
        try_files $uri $uri/ /index.html;
    }
}

3. Start Production Environment

# Start all VMs
vagrant up

# Check VM status
vagrant status

# SSH into specific VM
vagrant ssh gateway
vagrant ssh accounts
vagrant ssh textadv
vagrant ssh gameplays
vagrant ssh kafka

4. Deploy Services to VMs

Deploy to Accounts VM (192.168.50.12):

vagrant ssh accounts
cd /vagrant/accounts-service
mvn clean package
java -jar target/accounts-service-0.0.1-SNAPSHOT.jar --spring.profiles.active=prod

Deploy to TextAdv VM (192.168.50.13):

vagrant ssh textadv
cd /vagrant/textadventure-service
./gradlew bootJar
java -jar build/libs/textadventure-service-0.0.1-SNAPSHOT.jar --spring.profiles.active=prod

Deploy to Gameplays VM (192.168.50.14):

vagrant ssh gameplays
cd /vagrant/gameplays-service
javac -cp "lib/*" src/main/java/com/microservices/gameplays/*.java
java -cp "lib/*:src/main/java" com.microservices.gameplays.GameplaysApplication

Deploy Kafka to VM (192.168.50.15):

vagrant ssh kafka
cd /vagrant
docker-compose up -d

Configure Nginx Gateway (192.168.50.11):

vagrant ssh gateway
# Copy nginx config
sudo cp /vagrant/nginx/nginx.conf /etc/nginx/sites-available/default
sudo nginx -t
sudo systemctl restart nginx

# Deploy frontend
cd /vagrant/frontend
npm install
npm run build
sudo cp -r dist/* /var/www/html/

5. Access Production Environment

Add to your hosts file (/etc/hosts on Linux/Mac, C:\Windows\System32\drivers\etc\hosts on Windows):

192.168.50.11 api.local

Access the application:

Frontend: http://api.local
API Gateway: http://api.local/api/*
Direct VM Access:
- Accounts: http://192.168.50.12:8081
- TextAdv: http://192.168.50.13:8082
- Gameplays: http://192.168.50.14:8083

6. Production Environment Commands

# Stop all VMs
vagrant halt

# Destroy environment
vagrant destroy

# Reload specific VM
vagrant reload gateway

# View VM logs
vagrant ssh accounts -c "tail -f /var/log/accounts-service.log"

Service Details

Accounts Service

Technology: Spring Boot + Maven + PostgreSQL
Port: 8081
Database: PostgreSQL (accounts_db)
Features:
- User registration and authentication
- Account management
- Kafka events: account-created, account-updated

TextAdventure Service

Technology: Spring Boot + Gradle + MongoDB
Port: 8082
Database: MongoDB (textadventures_db)
Features:
- Adventure creation and management
- Questions and scenarios
- Links with account service
- Kafka events: adventure-created, adventure-updated

Gameplays Service

Technology: Pure Java + MySQL
Port: 8083
Database: MySQL (gameplays_db)
Features:
- Gameplay session storage
- Save game states
- Session management
- Kafka events: gameplay-saved, gameplay-completed

Frontend Application

Technology: Angular
Port: 4200
Features:
- Account management interface
- TextAdventure creation and editing
- Gameplay interface with save functionality
- Real-time updates via Kafka events

Docker Compose Services

Kafka Cluster

kafka-1: Port 9092
kafka-2: Port 9094
kafka-3: Port 9096

Databases

PostgreSQL: Port 5432 (accounts)
MongoDB: Port 27017 (textadventures)
MySQL: Port 3306 (gameplays)

Development Approaches

This project demonstrates different development approaches for microservices:

1. Maven Approach (Accounts Service)

Traditional Spring Boot with Maven
JPA/Hibernate for database operations
Standard dependency management

2. Gradle Approach (TextAdventure Service)

Modern build system with Gradle
MongoDB integration
Gradle-specific configurations

3. Pure Java Approach (Gameplays Service)

Minimal dependencies
Manual dependency management
Direct JDBC operations
Demonstrates raw Java capabilities

Kafka Event Flow

Account Created → TextAdventure Service (update user references)
Adventure Created → Gameplays Service (prepare gameplay sessions)
Gameplay Saved → Accounts Service (update user statistics)

API Endpoints

Accounts Service (8081)

POST /api/accounts - Create account
GET /api/accounts/{id} - Get account
PUT /api/accounts/{id} - Update account

TextAdventure Service (8082)

POST /api/adventures - Create adventure
GET /api/adventures - List adventures
GET /api/adventures/{id} - Get adventure details

Gameplays Service (8083)

POST /api/gameplays - Save gameplay
GET /api/gameplays/{userId} - Get user gameplays
PUT /api/gameplays/{id} - Update gameplay

Kafka Operations

Running Kafka Container Commands

Start the Kafka container:

docker-compose -f docker-compose.yml --env-file environment.env up -d
docker-compose -f docker-compose.yml --env-file environment.env down

Enter the Kafka container to execute commands:

docker exec -it kafka-microservices-kafka-1-1 /bin/bash
cd opt/bitnami/kafka/bin

Kafka Topic Management

List topics:

./kafka-topics.sh --list --bootstrap-server kafka-1:9090
# or externally
docker exec -it kafka-microservices-kafka-1-1 kafka-topics.sh --list --bootstrap-server localhost:9092

Create a topic:

./kafka-topics.sh --create --topic test-topic --bootstrap-server kafka-1:9090,kafka-2:9090,kafka-3:9090

Produce and Consume Messages

Produce messages:

docker exec -it kafka-microservices-kafka-1-1 kafka-console-producer.sh --topic test-topic --bootstrap-server kafka-1:9090

Consume messages (in another terminal):

docker exec -it kafka-microservices-kafka-1-1 kafka-console-consumer.sh --topic test-topic --bootstrap-server kafka-1:9090 --from-beginning

Note: Use internal bootstrap server addresses (kafka-1:9090) when running commands from inside containers, and external addresses (localhost:9092) when accessing from the host machine.

Environment Configuration

Database Credentials

PostgreSQL: postgres/postgres
MongoDB: root/example
MySQL: root/password

Kafka Configuration

Bootstrap servers: localhost:9092,localhost:9094,localhost:9096
Consumer groups: service-specific groups
JSON serialization for all services

Testing the System

1. Test Kafka Integration

# Create a test topic
docker exec -it kafka-microservices-kafka-1-1 kafka-topics.sh --create --topic test --bootstrap-server localhost:9092

# List topics
docker exec -it kafka-microservices-kafka-1-1 kafka-topics.sh --list --bootstrap-server localhost:9092

2. Test Service Communication

# Create an account
curl -X POST http://localhost:8081/api/accounts -H "Content-Type: application/json" -d '{"username":"testuser","email":"[email protected]","password":"password"}'

# Create an adventure
curl -X POST http://localhost:8082/api/adventures -H "Content-Type: application/json" -d '{"title":"Test Adventure","description":"A test adventure"}'

# Save gameplay
curl -X POST http://localhost:8083/api/gameplays -H "Content-Type: application/json" -d '{"userId":1,"adventureId":1,"gameState":"started"}'

Monitoring and Logs

View Kafka Topics

docker exec -it kafka-microservices-kafka-1-1 kafka-console-consumer.sh --topic account-created --from-beginning --bootstrap-server localhost:9092

Service Logs

# Docker services
docker-compose logs -f kafka-1
docker-compose logs -f postgres
docker-compose logs -f mongodb
docker-compose logs -f mysql

Scaling and Production

Horizontal Scaling

Each microservice can be scaled independently
Kafka partitioning supports parallel processing
Database sharding can be implemented per service

Production Considerations

External configuration management
Service discovery (Eureka/Consul)
API Gateway (Spring Cloud Gateway)
Distributed tracing (Zipkin/Jaeger)
Container orchestration (Kubernetes)

Troubleshooting

Common Issues

Port conflicts: Ensure ports 8081-8083, 4200, 9092, 9094, 9096, 5432, 27017, 3306 are available
Kafka connectivity: Verify all Kafka brokers are running
Database connections: Check database containers are healthy
Java version: Ensure Java 24 is installed and configured

Reset Environment

# Stop all services
docker-compose down -v

# Clean up containers and volumes
docker system prune -f
docker volume prune -f

# Restart infrastructure
docker-compose up -d

Contributing

Fork the repository
Create feature branches for each microservice
Follow the established patterns for each build system
Ensure Kafka integration for new features
Update documentation and tests

License

This project is licensed under the MIT License - see the LICENSE file for details.

Name		Name	Last commit message	Last commit date
Latest commit History 5 Commits
accounts-service		accounts-service
.gitignore		.gitignore
KAFKA-CONFIG.md		KAFKA-CONFIG.md
KAFKA-README.md		KAFKA-README.md
LICENSE		LICENSE
MICROSERVICES-LOCALLY.md		MICROSERVICES-LOCALLY.md
README.md		README.md
ROADMAP.md		ROADMAP.md
docker-compose.yml		docker-compose.yml
environment.env		environment.env
start_single_kafka_broker.md		start_single_kafka_broker.md

Uh oh!

License

Uh oh!

all-an/Apache-Kafka-for-Event-Driven-Spring-Boot-Microservices

Folders and files

Latest commit

History

Repository files navigation

Event-Driven Spring Boot Microservices with Apache Kafka

Architecture Overview

Microservices

Infrastructure

Deployment Options

Option 1: Development Setup (Docker Compose)

Option 2: Production Simulation (Vagrant + Nginx)

Quick Start

Prerequisites

For Development Setup (Option 1)

For Production Simulation (Option 2)

Setup Instructions

Option 1: Development Setup

1. Start Infrastructure

2. Start Microservices

Accounts Service (Maven)

TextAdventure Service (Gradle)

Gameplays Service (Pure Java)

Frontend (Angular)

Option 2: Production Simulation (Vagrant + Nginx)

1. Create Vagrant Configuration

2. Create Nginx Configuration

3. Start Production Environment

4. Deploy Services to VMs

Deploy to Accounts VM (192.168.50.12):

Deploy to TextAdv VM (192.168.50.13):

Deploy to Gameplays VM (192.168.50.14):

Deploy Kafka to VM (192.168.50.15):

Configure Nginx Gateway (192.168.50.11):

5. Access Production Environment

6. Production Environment Commands

Service Details

Accounts Service

TextAdventure Service

Gameplays Service

Frontend Application

Docker Compose Services

Kafka Cluster

Databases

Development Approaches

1. Maven Approach (Accounts Service)

2. Gradle Approach (TextAdventure Service)

3. Pure Java Approach (Gameplays Service)

Kafka Event Flow

API Endpoints

Accounts Service (8081)

TextAdventure Service (8082)

Gameplays Service (8083)

Kafka Operations

Running Kafka Container Commands

Kafka Topic Management

Produce and Consume Messages

Environment Configuration

Database Credentials

Kafka Configuration

Testing the System

1. Test Kafka Integration

2. Test Service Communication

Monitoring and Logs

View Kafka Topics

Service Logs

Scaling and Production

Horizontal Scaling

Production Considerations

Troubleshooting

Common Issues

Reset Environment

Contributing

License

About

Resources

License

Packages