Initial prototype

[danchild]

---

Forge Observability Plugin — Core Architecture Proposal

This PR establishes the foundational architecture, tooling, and data pipeline design for the Forge
Observability plugin. It is intended as a structural proposal for review and discussion, not a
feature-complete implementation.

The placeholder data models, example API endpoints, and seeded metrics are not recommendations — they exist
solely to exercise the full stack end-to-end and give reviewers something concrete to react to. The specific
metrics, schemas, and API shapes that matter for Forge Observability will be defined in subsequent work
once this foundation is validated.

---

What this establishes

Plugin structure — The package integrates cleanly into the broader Forge CLI namespace as forge
observability, contributing its own subcommand without requiring changes to the host CLI. The same plugin
pattern can be replicated by future Forge modules.

Data pipeline architecture (medallion) — Four dlt source pipelines (Langfuse, GitHub, JIRA, Prometheus) load
raw data into bronze tables concurrently via asyncio.gather. After each ingestion round, a dbt project
rebuilds silver and gold views on top of the raw data. This bronze → silver → gold separation gives us a
clean contract between raw ingestion and analytics-ready output — each layer can evolve independently.

Datastore-agnostic design — The repository layer (repository/repository.py) is a thin SQLAlchemy Core
abstraction. The API never speaks directly to the external datastore — it only constructs select() statements and hands
them to Repository.query(). Swapping the backing datastore is a one-line DSN change. ClickHouse is the
reference implementation here, but nothing in the application layer is coupled to it.

Modern data stack tooling rationale — dlt and dbt have emerged as reliable, widely-adopted primitives in the
modern data stack. dlt handles schema inference, incremental loading, and connector maintenance so we don't
have to; dbt provides a principled transformation layer with lineage, testing hooks, and a macro system
(see is_source_available()) that lets models degrade gracefully when upstream sources haven't loaded yet.
Adopting both now means we're building on a foundation with strong community support, rather than bespoke
pipeline glue that we'd have to maintain ourselves.

Containerized dev environment — The API and worker services are independently containerized and composed
with ClickHouse in devtools/, making it straightforward to run the full stack locally or adapt the
Dockerfiles for any target deployment environment.

Test harness — The test suite mocks Repository at the FastAPI dependency boundary, so API tests run without
a live datastore. Pipeline helper logic is tested in isolation. This pattern keeps the test suite fast and
reinforces the value of the repository abstraction.

---

What this is not

• A recommendation for which metrics to expose or track
• A final data model — table shapes and silver/gold views are illustrative
• A production-ready pipeline — intervals, error handling, and retry behavior will need hardening
• A commitment to ClickHouse — it's a practical default for columnar analytics, but the abstraction boundary
  is intentional

---

What comes next

With the piping in place, future work can focus on: defining the metrics that actually matter for SDLC
observability, populating the gold layer with meaningful KPIs, and wiring the API to real Forge workflows —
without revisiting the ingestion architecture.