Narwhal

Azure-based log aggregation pipeline for Docker containers.

---

Architecture

Project Structure

├── cmd/
│   ├── ingest/            # HTTP ingestion server
│   ├── processor/         # Event Hub consumer → Cosmos DB writer
│   ├── dashboard/         # Read-only log query API
│   └── alerter/           # Error rate spike detector
├── internal/
│   ├── handler/           # HTTP handlers (ingest + query)
│   ├── store/             # Cosmos DB client
│   ├── publish/           # Event Hub publisher
│   ├── alert/             # Alert logic + webhook sender
│   └── middleware/        # Logging & metrics middleware
├── pkg/
│   ├── models/            # Shared data types
│   ├── parse/             # Log entry parser
│   └── errors/            # Error definitions
├── configs/               # Fluent Bit configuration
├── docker/                # Dockerfiles for all services
├── infra/                 # Terraform (Azure infrastructure)
└── scripts/               # Load testing

Development

Prerequisites

• Go 1.25+
• Azure CLI
• Terraform

Azure Infrastructure

make azure-up        # provision Event Hub + Cosmos DB
make env-fetch       # pull connection strings into .env
make azure-down      # tear down resources

Run Services

Start each in a separate terminal:

make run-ingest      # HTTP ingest on :8080
make run-processor   # Event Hub → Cosmos DB
make run-dashboard   # Log query API on :8081
make run-alerter     # Error rate polling
make run-fluent-bit  # Captures all Docker container logs

Collect Logs from Other Projects

Start Fluent Bit with make run-fluent-bit. It tails all Docker container logs on the host automatically — any docker compose project running on the same machine will have its logs captured and forwarded to narwhal.

Test

make test            # run all tests
make curl-single     # send a test log
make curl-logs       # query logs from dashboard
make loadtest        # 1000 requests, 50 concurrent

Query Logs

# all recent logs
curl -s 'http://localhost:8081/api/logs?limit=10'

# filter by container, level, time range
curl -s 'http://localhost:8081/api/logs?container_id=my-app&level=error&since=2026-01-01T00:00:00Z'

Environment Variables

See docs/configuration.md for all env vars per service. Quick start:

make env-fetch       # pulls all required vars from Terraform into .env

Docker

make docker-build    # build all service images