Observability Stack with Grafana LGTM

A complete observability stack demo featuring Grafana, Loki, Tempo, and Prometheus with OpenTelemetry instrumentation.

Features

• Grafana: Unified observability dashboard
• Prometheus: Metrics collection and storage with remote write receiver
• Tempo: Distributed tracing with metrics generator for service graphs
• Loki: Log aggregation
• Vector: Log shipping from containers to Loki
• OpenTelemetry: Full instrumentation with trace ID injection into logs
• React Frontend: Sample application to generate traffic
• Node.js Backend: Express API with auto-instrumentation
• PostgreSQL: Database for demo application

Architecture

┌─────────────┐
│   Frontend  │
│   (React)   │
└──────┬──────┘
       │
       ▼
┌─────────────┐      ┌──────────────┐
│   Backend   │─────▶│  PostgreSQL  │
│  (Express)  │      └──────────────┘
└──────┬──────┘
       │
       ├──────────▶ Tempo (OTLP traces)
       │
       └──────────▶ Prometheus (metrics)

┌─────────────┐
│   Vector    │──────▶ Loki (logs with trace_id)
└─────────────┘

Tempo ──────────────▶ Prometheus (trace metrics via remote write)

       ┌─────────────┐
       │   Grafana   │
       │  Dashboard  │
       └─────────────┘
            │
            ├──── Prometheus
            ├──── Tempo
            └──── Loki

Quick Start

Prerequisites

• Docker and Docker Compose installed
• Ports available: 3000 (Grafana), 3001 (Frontend), 4000 (Backend), 9090 (Prometheus), 3200 (Tempo), 3100 (Loki), 5432 (PostgreSQL)

Start the Stack

# Start all services
docker-compose up -d

# View logs
docker-compose logs -f

# Stop all services
docker-compose down

# Stop and remove volumes (clean slate)
docker-compose down -v

Accessing the Services

• Frontend Application: http://localhost:3001
• Grafana Dashboard: http://localhost:3000
  • Pre-configured with datasources and dashboard
  • No login required (anonymous access enabled)
• Prometheus: http://localhost:9090
• Tempo: http://localhost:3200
• Backend API: http://localhost:4000
  • Health: http://localhost:4000/health
  • Metrics: http://localhost:4000/metrics

Using the Demo

1. Open the frontend at http://localhost:3001
2. Interact with the application:
   • Create, view, and delete users
   • Click "Trigger Error" to generate error traces and logs
   • Click "Slow Request" to generate slow traces
3. Open Grafana at http://localhost:3000
4. Navigate to "Observability Stack Dashboard"
5. Explore:
   • Service Graph: View service relationships generated by Tempo
   • Metrics: HTTP request rates, response times, status codes
   • Logs: View logs with embedded trace IDs
   • Traces: Click on trace IDs in logs to jump to traces
   • Correlations: Jump between metrics, logs, and traces

Key Features Demonstrated

Trace ID Injection

Logs automatically include trace_id from OpenTelemetry context, enabling correlation between logs and traces.

Service Graph

Tempo's metrics generator creates service relationship metrics and pushes them to Prometheus via remote write.

Exemplars

Metrics in Prometheus include exemplar trace IDs, allowing you to jump from a metric spike to example traces.

Three Pillars Correlation

• Click a metric spike → Jump to traces
• Click a trace → Jump to related logs
• Click a log → Jump to the trace

Configuration Files

• docker-compose.yml: Main orchestration file
• prometheus/prometheus.yml: Prometheus config with remote write enabled
• tempo/tempo.yaml: Tempo config with metrics generator and OTLP receivers
• loki/loki.yaml: Loki configuration
• vector/vector.yaml: Vector log shipping configuration
• grafana/provisioning/: Pre-configured datasources and dashboards
• backend/src/tracing.js: OpenTelemetry SDK configuration
• backend/src/logger.js: Winston logger with trace context injection

API Endpoints

Users API

• GET /api/users - Get all users
• GET /api/users/:id - Get user by ID
• POST /api/users - Create user (body: {name, email})
• PUT /api/users/:id - Update user (body: {name, email})
• DELETE /api/users/:id - Delete user

Testing Endpoints

• GET /api/error - Trigger an error for testing
• GET /api/slow - Simulate a slow request (3 seconds)
• GET /health - Health check
• GET /metrics - Prometheus metrics

Observability Stack Details

OpenTelemetry Instrumentation

The backend uses OpenTelemetry with:

• Auto-instrumentation for HTTP, Express, and PostgreSQL
• OTLP HTTP exporter to Tempo
• Trace context propagation
• Span attributes and events

Log-Trace Correlation

Winston logger is configured to:

1. Extract trace context from active span
2. Inject trace_id and span_id into logs
3. Output JSON logs for Vector to parse
4. Vector ships logs to Loki with trace_id labels

Metrics Generation

• Prometheus scrapes application metrics
• Tempo generates span metrics and service graph metrics
• Tempo pushes trace metrics to Prometheus via remote write
• Service graphs are built from these metrics

Troubleshooting

Services not starting

# Check service status
docker-compose ps

# View specific service logs
docker-compose logs backend
docker-compose logs tempo

No traces appearing

• Verify Tempo is running: docker-compose logs tempo
• Check backend OTLP endpoint: OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo:4318
• Look for trace exports in backend logs

No logs in Loki

• Check Vector is running: docker-compose logs vector
• Verify Vector can reach Loki
• Check container log format is JSON

Service graph not showing

• Wait a few minutes for metrics to generate
• Ensure Tempo metrics generator is enabled
• Check Prometheus receives remote write data: http://localhost:9090

Development

Backend Development

cd backend
npm install
npm run dev

Frontend Development

cd frontend
npm install
npm start

Tech Stack

• Frontend: React, Axios
• Backend: Node.js, Express, Winston, Prom-client
• Instrumentation: OpenTelemetry SDK & Auto-instrumentations
• Database: PostgreSQL
• Observability: Grafana, Prometheus, Tempo, Loki, Vector

License

MIT