Real-time pipeline for log analysis with AI
A simple and simplified forward-proxy-caching-IDS-AI-log-analyzer comprehensive system, made for self-educational purpose. Long story short: with the help of ChatGPT I found some ideas to practice my coding skills (!) and learn something new about networks, data, AI...
LLMs (mostly ChatGPT and Claude) have been used as a support to design the architecture, generate/study parts of the code and to write the documentation.
The whole project is divided in modules. All the documentation in the 'docs' folder:
| # | Name | Description | Languages | Status |
|---|---|---|---|---|
| 00 | Overview | Architecture and instructions | - | Completed |
| 01 | Proxy | A forward proxy | Go | Completed |
| 02 | ETL | Log ETL tool | Rust | Ongoing |
| 03 | Agent | AI-based log analysis | Python | Pending |
Every module has its own Dockerfile in the docker folder. Container orchestration is done via Kubernetes using the files in the k8s folder. Communication between modules is done via files and Kafka topics. More about general architecture and usage in the Overview.
My beloved wife and my wonderful kids, who let me some spare time here and there to purse my hobbies.
- RFC 2616 on HTTP/1.1 Protocol, especially section 9
- Arrow format documentation
- Go by Example contains all you need to learn/refresh Go
- Then the http package docs is mandatory
- Alex Rios, System Programming Essentials with Go, Packt Publishing 2024
- The Rust Programming Language
- docs.rs for all the crates documentation
Subject to changes.
| Task | Module | Status |
|---|---|---|
| Implement a forward proxy server | Proxy | Completed |
| Implement a log ETL tool | ETL | Completed |
| Implement Kafka communication | - | In Progress |
| Containerize the modules with Docker | - | In Progress |
| Orchestrate containers with Kubernetes | - | In Progress |
| Implement AI-based log analysis | Agent | Pending |
| Implement IDS functionalities | IDS | Pending |
| Implement blocklist functionalities | Proxy | Pending |
| Implement caching functionalities | Proxy | Pending |
This project is under Apache 2.0 License.