Phase 9: Observability & Monitoring Architecture

[vimal-java-dev]

Since we're running:

• Spring Boot 3.4.10
• Java 21
• Docker on VPS
• CI/CD via GitHub → GHCR
• Nginx reverse proxy
• In-memory rate limiting

We’ll design a clean, realistic observability stack that is appropriate for a portfolio project but architecturally correct.

---

Phase 9: Observability & Monitoring Architecture

You want three pillars:

1. Metrics (quantitative signals)
2. Logs (event data)
3. Health & Uptime

Optionally:
4. Distributed tracing (advanced tier)

[vimal-tech-starter]

Assign me.

[vimal-tech-starter]

Step 1: Enable Metrics (Local Machine)

Everything starts locally, then I push to GitHub.

1️⃣ Add Dependencies

• Opened my local project → pom.xml.

• dependencies: actuator, prometheus.

---

2️⃣ Configure Actuator

• src/main/resources/application-prod.yml
• This enables the Prometheus metrics endpoint.

---

3️⃣ Test Locally (VERY IMPORTANT)

1. Before pushing anything, verify locally using Docker.
2. Before run docker, change the profile from dev to prod.
3. Update the docker-compose.yml file for production
4. Make a separate .env
5. -> .env.dev .env.prod at root of project
6. Add .env.prod in docker compsoe file.
7. Docker will automatically load variables from .env.
   ===================

• Why This Pattern Is Better
• This is a standard DevOps pattern:
• application-prod.yml → references variables
• .env / secrets manager → supplies values

---

4️⃣ When Running Locally (Dev) [ Note: not implemented yet]

• Load .env.dev before starting Spring Boot.
• Example (PowerShell / Git Bash):
• export $(cat .env.dev | xargs)
• mvn spring-boot: run -Dspring-boot.run.profiles=dev

---

5️⃣ Rebuild Containers After Fix

• docker compose down

• docker compose up -d --build

• Then check logs:

• docker logs vimaltech-contact-api

• Then Verify Metrics -> curl http://localhost:8080/actuator

• Prometheus metrics output. -> curl http://localhost:8080/actuator/prometheus

---

Summary

1. Do Step 5 now because you are validating the Docker production environment.
2. Step 4 is only needed when running:
3. mvn spring-boot:run
4. outside Docker.

---

6️⃣ Next Step (Real Observability)

Implementation Step
I added two containers to your docker-compose.yml:
1️⃣ Prometheus
2️⃣ Grafana
and a small prometheus.yml configuration file.

================================
Docker Network
├─ Spring Boot API
│ └─ /actuator/prometheus
│
├─ PostgreSQL
│
├─ Prometheus
│ └─ scrapes metrics every 15s
│
└─ Grafana
└─ visual dashboards

========================
The goal is to extend my stack from:

Spring Boot API → /actuator/prometheus
PostgreSQL

to a complete monitoring pipeline:

Spring Boot API
↓
/actuator/prometheus
↓
Prometheus (scrapes metrics)
↓
Grafana (visual dashboards)

---

1️⃣ Project Structure

Create two new folders in your project root:

vimaltech-contact-api
│
├─ docker-compose.yml
├─ prometheus
│ └─ prometheus.yml
│
├─ grafana
│ └─ provisioning (optional later)
│
└─ src/

---

2️⃣ Create Prometheus Config

• Create:
• prometheus/prometheus.yml

---

3️⃣ Update docker-compose.yml

• You only need to add two services.
• Prometheus:
• Grafana:

---

4️⃣ Start the Monitoring Stack

• docker compose down
• docker compose up -d --build
• docker ps

---

5️⃣ Verify Prometheus

6️⃣ Setup Grafana

7️⃣ Your Final Architecture

My system now looks like a real production observability stack:

Client
↓
Nginx (VPS)
↓
Spring Boot API
↓
Micrometer
↓
/actuator/prometheus
↓
Prometheus
↓
Grafana

---

[vimal-tech-starter]

GitHub CI/CD Builds Image

Your GitHub Actions pipeline should now:

1. Build the JAR
2. Build Docker image
3. Push image to GHCR

Example result:

• ghcr.io/vimaltech/contact-api:latest

Wait until the pipeline finishes successfully.

[vimal-tech-starter]

Done below at the production level [VPS]

1. contact-api, postgres at a separate Docker compose.
2. Monitoring - Prometheus, alert, Grafana, separate Docker Compose.
3. Real DevOps configuration

• Prometheus is working (targets UP)
• Grafana connected to Prometheus
• Metrics like http_server_requests_seconds_count are visible
• Dashboard created: "Vimal Tech Contact API Observability."
• Basic panels working (rate, increase)

[vimal-java-dev]

Done below at the production level [VPS]

1. contact-api, postgres at a separate Docker compose.
2. Monitoring - Prometheus, alert, Grafana, separate Docker Compose.
3. Real DevOps configuration

• Prometheus is working (targets UP)
• Grafana connected to Prometheus
• Metrics like http_server_requests_seconds_count are visible
• Dashboard created: "Vimal Tech Contact API Observability."
• Basic panels working (rate, increase)

👍 Good News

You’ve already completed the hardest part:

✔ Docker networking
✔ Prometheus scraping
✔ Alerting
✔ Grafana integration

Now it’s just refinement and presentation.

[vimal-tech-starter]

Check my PR #33.

[vimal-java-dev]

Check my PR #33.

PR #33 merged and closed.

[vimal-tech-starter]

🚀 FINAL CHECKLIST For the Grafana panel for a specific row in the "Vimal Tech Contact API Observability" dashboard.

Done Below.

✅ 3 panels in Row 1 aligned perfectly
✅ No panel wrapping to next line
✅ Units set everywhere
✅ Threshold colors visible
✅ No empty panels
✅ Clean titles

[vimal-tech-starter]

✔ Naming Consistency

Use EXACT format everywhere:

• API Request Rate (RPS)
• Error Rate (%)
• API Latency (p95)
• Total Requests (5m)
• Heap Usage (%)
• CPU Usage (%)
• Live Threads
• Service Health

[vimal-java-dev]

Phase 9: Observability & Monitoring Architecture is completed.