Modern distributed systems generate large amounts of log data from various sources, making effective analysis essential for monitoring, security, and optimization. This project introduces a custom distributed log analysis framework based on the MapReduce paradigm, allowing scalable log processing and aggregation across multiple nodes. Our system extracts key data such as the most active IP addresses and endpoints using a microservices architecture, consisting of a Java Spring Boot API gateway, Go-based analyzer, MongoDB, and a reliable messaging pipeline using RabbitMQ. The framework has been tested using generated logs that are similar to those found in the real world, and it provides an interactive dashboard. The dashboard focuses on extracting the most active IP addresses or endpoints from server logs. This allows organizations to gain valuable insights from their distributed system's data.
- Docker and Docker Compose
- Git
git clone https://github.com/yourusername/Distributed-Log-Analysis-Framework.git
cd Distributed-Log-Analysis-FrameworkNavigate to the docker directory and start all services:
cd docker
docker-compose up -d --buildCheck that all containers are running:
docker ps- URL: http://localhost:8081
- Login: admin
- Password: admin
- Port: 27018
- Login: admin
- Password: admin
- URL: http://localhost:15672
- Login: admin
- Password: admin
- Server 1: http://localhost:8001
- Server 2: http://localhost:8002
- Server 3: http://localhost:8003
- Metrics endpoint: /metrics
- URL: http://localhost:8090
- Metrics endpoint: /metrics
- URL: http://localhost:8091
- Metrics endpoint: /metrics
- URL: http://localhost:3000
- Login: admin
- Password: admin
When you're done, you can stop all services with:
docker compose down -vFor more detailed information about specific components, refer to the documentation section below.