Eventually Consistent Highly Scalable System

A production-ready, highly scalable, eventually consistent system demonstrating modern microservices architecture with event-driven design patterns.

🚀 Quick Start

# Clone the repository
git clone <repository-url>
cd eventually-consistent-highly-scalable-system

# Start all services
docker compose up -d

# Access the web application
# Option 1: Port-based access
open http://localhost:3001

# Option 2: Domain-based access (add to /etc/hosts)
echo "127.0.0.1 app.eventualconsistent.local" >> /etc/hosts
open http://app.eventualconsistent.local:8080

# Access monitoring tools
open http://localhost:3000  # Grafana
open http://localhost:16686 # Jaeger

🔒 HTTPS Setup

To run the web application using HTTPS with SSL/TLS encryption:

Automated HTTPS Setup

# Run the automated HTTPS setup script
./setup-https.sh

This script will:

✅ Generate SSL certificates for both frontend and backend
✅ Configure HAProxy for HTTPS termination
✅ Update /etc/hosts with the local domain
✅ Build and start all services with HTTPS enabled
✅ Perform health checks to verify HTTPS is working

Manual HTTPS Setup

If you prefer to setup HTTPS manually:

# 1. Generate SSL certificates
chmod +x scripts/generate-ssl-cert.sh scripts/generate-backend-ssl.sh
./scripts/generate-ssl-cert.sh      # Frontend SSL certificates
./scripts/generate-backend-ssl.sh   # Backend SSL certificates

# 2. Create Docker volume for certificates
docker volume create ssl-certs

# 3. Add domain to /etc/hosts (optional for domain-based access)
echo "127.0.0.1 app.eventualconsistent.local" | sudo tee -a /etc/hosts

# 4. Start services
docker compose down
docker compose up -d --build

# 5. Wait for services to start
sleep 30

# 6. Verify HTTPS is working
curl -k https://localhost:8443/health

HTTPS Access Points

Once HTTPS is configured, access your services at:

Service	HTTPS URL	Purpose
Web Application	https://localhost:3001	React frontend
API Gateway	https://localhost:8443	HAProxy load balancer
Domain Access	https://app.eventualconsistent.local:8443	Domain-based access
Keycloak Admin	https://localhost:8443/admin	Authentication server

Development Mode with HTTPS

To run the React app in development mode with HTTPS:

cd web_app
npm start  # Will use HTTPS if certificates are configured

Certificate Information

Certificate Type: Self-signed certificates for development
Certificate Location: Docker volume ssl-certs
Security Warning: You'll see browser warnings for self-signed certificates
- Click "Advanced" → "Proceed to localhost (unsafe)" to continue
Production: Replace with CA-signed certificates for production use

Troubleshooting HTTPS

# Check if HTTPS services are running
docker compose ps

# Test HTTPS connectivity
curl -k https://localhost:8443/health

# View HAProxy logs for SSL issues
docker compose logs load-balancer

# Regenerate certificates if needed
./scripts/generate-ssl-cert.sh
./scripts/generate-backend-ssl.sh

📚 Documentation Index

Core Documentation

Document	Description	Key Files
Architecture	System design, CQRS patterns, and service communication	`docker-compose.yml`, `load_balancer/haproxy.cfg`
Deployment Guide	Setup, configuration, and getting started	`scripts/`, `config/`
API Reference	Complete API documentation with examples	`_service/src/main/java//controller/`

Operations & Security

Document	Description	Key Files
Monitoring & Observability	Metrics, alerting, and production readiness	`monitoring/`, `prometheus.yml`
Authentication & Authorization	Keycloak integration and security patterns	`auth/`, `realm-export.json`
Security Guide	Security concerns and best practices	`config/ssl/`, `load_balancer/haproxy.cfg`
Incident Response	Security incident response procedures	`scripts/security-scan.sh`, `monitoring/alert-rules.yml`
Security Implementation Summary	Complete security implementation status	`config/`, `ssl-cert-generator service`

🏗️ System Overview

This system implements a CQRS (Command Query Responsibility Segregation) pattern with:

Event-driven architecture using RabbitMQ (config/rabbitmq/)
Keycloak-based authentication with enterprise-grade security (auth/)
Comprehensive monitoring with distributed tracing (monitoring/)
High availability and horizontal scaling capabilities (load_balancer/)

Key Components

Component	Port	Purpose	Configuration
React Web App	3001	Frontend user interface	`web_app/`
HAProxy	8080/8443	Load balancer and API gateway	`load_balancer/haproxy.cfg`
Write Layer	8081/8083	Event-driven write operations	`write_layer/`
Public API	8082	Read operations with caching	`public_api/`
Coupon Service	8086	Business logic microservice	`coupon_service/`
Keycloak	8090	Authentication server	`auth/realm-export.json`

Infrastructure Services

Service	Port	Purpose	Configuration
RabbitMQ	5672/15672	Message broker	`config/rabbitmq/rabbitmq.conf`
Redis	6379	Cache layer	`config/redis.conf`
DynamoDB	8000	NoSQL database	`scripts/init-dynamodb.sh`
Kafka	9092	Event streaming	`docker-compose.yml`
PostgreSQL	5432	Keycloak database	`auth/postgres-init.sql`

🔄 Architecture Patterns

Write Operations (Event-Driven)

Client → HAProxy (TLS) → Write Layer → RabbitMQ → Services → Database

See: write_layer/src/main/java/com/example/writelayer/controller/WriteController.java

Read Operations (Cache-First)

Client → HAProxy (TLS) → Service → Redis Cache → Database (fallback)

See: public_api/src/main/java/com/example/publicapi/controller/PublicApiController.java

🔐 Security Features

This system implements enterprise-grade security with comprehensive protection:

🛡️ Infrastructure Security

TLS/SSL Encryption: HTTPS enabled with HAProxy SSL termination (ssl-cert-generator service)
Network Segmentation: Internal and backend networks with secure communication (docker-compose.yml networks)
Service Mesh: mTLS configured for service-to-service communication (load_balancer/haproxy.cfg)
Secure Service Discovery: Services communicate over protected internal networks (config/)

🔒 Application Security

Input Validation: Comprehensive validation and sanitization for all inputs (*/src/main/java/*/service/InputValidationService.java)
Authentication: Keycloak-based JWT authentication with role-based access control (auth/realm-export.json)
Authorization: Fine-grained permissions and secure API endpoints (*/src/main/java/*/config/SecurityConfig.java)
Session Management: Secure JWT token lifecycle with automatic refresh (auth/)

🗄️ Data Security

Encryption at Rest: Database and cache encryption configured (config/redis.conf)
Encryption in Transit: TLS for all service communications (SSL certificates volume)
Secrets Management: Secure credential storage and rotation (config/)
Access Controls: Database and service-level security policies (auth/postgres-init.sql)

📊 Security Monitoring

Real-time Alerts: Prometheus-based security event monitoring (monitoring/alert-rules.yml)
Audit Logging: Comprehensive security event tracking (monitoring/prometheus.yml)
Incident Response: Automated detection and response procedures (scripts/)
Penetration Testing: Regular automated security assessments (scripts/security-scan.sh)

🔧 Security Tools

# Complete security setup
./scripts/setup-security.sh

# Generate SSL certificates
./scripts/setup-ssl.sh

# Manage secrets securely
./scripts/secrets-manager.sh

# Run security vulnerability scan
./scripts/security-scan.sh

# Automated penetration testing
./scripts/penetration-test.sh

🔐 Authentication

Default users for testing:

Admin: admin / admin
User: testuser / testuser

Access points:

Web App (Port): http://localhost:3001
Web App (Domain): http://app.eventualconsistent.local:8080
Keycloak Admin: http://localhost:8090/auth/admin

Configuration: auth/realm-export.json

📊 Monitoring

Tool	URL	Purpose	Configuration
Grafana	http://localhost:3000	Dashboards	`monitoring/grafana-dashboards.yml`
Jaeger	http://localhost:16686	Distributed tracing	`docker-compose.yml`
Prometheus	http://localhost:9090	Metrics	`monitoring/prometheus.yml`
RabbitMQ	http://localhost:15672	Message queues	`config/rabbitmq/rabbitmq.conf`

🧪 Testing

# Health check all services
./scripts/production-health-check.sh

# Test event-driven operations
./scripts/test-idempotency.sh

# Performance testing
./scripts/performance-test.sh

# Security vulnerability scanning
./scripts/security-scan.sh

# Automated penetration testing
./scripts/penetration-test.sh

# Complete security setup and validation
./scripts/setup-security.sh

🐳 Docker Services

# View service status
docker compose ps

# View logs
docker compose logs -f [service-name]

# Scale services
docker compose up -d --scale write-layer=3

🔧 Development

For detailed development instructions, see:

Architecture Documentation - Understanding the system design
Deployment Guide - Setting up development environment
API Reference - Using the APIs

📈 Production Readiness

This system includes:

✅ Health checks and monitoring
✅ Distributed tracing with Jaeger
✅ Enterprise security with comprehensive protection
✅ Backup and recovery procedures
✅ Performance testing capabilities
✅ High availability patterns
✅ Incident response procedures
✅ Penetration testing framework

For production deployment details, see Monitoring & Observability.

🛡️ Security Checklist

All security requirements have been implemented:

Infrastructure Security ✅

Network segmentation implemented
TLS encryption for all communications
Service mesh with mTLS configured
Secure service discovery
Regular security updates and patches

Application Security ✅

Data Security ✅

Monitoring and Incident Response ✅

Compliance and Governance ✅

Security policies and procedures
Regular security training
Compliance monitoring
Risk assessment and management
Third-party security assessments

🤝 Contributing

Fork the repository
Create a feature branch
Follow the architecture patterns documented in docs/
Add tests and documentation
Submit a pull request

📄 License

MIT License

For detailed information on any aspect of the system, please refer to the specific documentation files in the docs/ directory.

Name		Name	Last commit message	Last commit date
Latest commit History 8 Commits
.cursor/rules		.cursor/rules
auth		auth
cache-sync-service		cache-sync-service
config		config
coupon_service		coupon_service
docs		docs
load_balancer		load_balancer
monitoring		monitoring
order_service		order_service
public_api		public_api
scripts		scripts
shared-model		shared-model
ssl-certs		ssl-certs
test		test
web_app		web_app
write_layer		write_layer
.gitignore		.gitignore
README.md		README.md
docker-compose.test.yml		docker-compose.test.yml
docker-compose.yml		docker-compose.yml
haproxy.cfg		haproxy.cfg
mock_coupon_server.py		mock_coupon_server.py
pom.xml		pom.xml
setup-https.sh		setup-https.sh
test_coupon_flow.sh		test_coupon_flow.sh
test_coupon_functionality.sh		test_coupon_functionality.sh

jbdoster/eventually-consistent-highly-scalable-system

Folders and files

Latest commit

History

Repository files navigation