Export our metrics using OpenTelemetry

[leonhandreke]

Seems to be what all the cool kids are doing these days.

Must:

• the stuff currently in /catalog

Nice to have:

• other standard things? Latency by endpoints and backend maybe? Does Elastic maybe have something built-in?
• OTLP push

Stretch:

• Traces and whatever other magic is in there - I saw some really cool stuff at a demo of https://github.com/apognu/motiva recently

[dimoschi]

@leonhandreke I'd be more than happy to take this on and create a PR. Let me know if that's ok!

[leonhandreke]

Hey @dimoschi that sounds super nice! 😍 Would also love to hear how you'd be using this yourself!

I'm not too deep into the whole ecosystem, so not all of this might make sense, but here is my initial thinking:

• Let's instrument this thing in the most standard, basic and low-code way possible. I think all advanced advanced features that require more work from our side can come later as we need them. Let's get something basic rolling first :)
• I see Elastic should already have OpenTelemetry built in
• The OpenTelemetry Python SDK seems to have a ready-to-go thing for FastAPI.
• There seems to be a lot of standard configuration variables. Is there a way we can let our instrumentation be configurable like every other app under the sun with only a single line? Can we make it read the standard OTEL_CONFIG_FILE?
• Adding a /metrics endpoint seems like another standard thing that we would want to have - or not? Maybe the modern way to do this is to only support the OpenTelemetry push paradigm?
• Let's keep the PRs small-ish and easily reviewable :)

Let me know what your thinking around this is. Also happy to jump on a call to exchange thoughts :)

[dimoschi]

Hey @leonhandreke, thanks for the detailed thoughts! Here's my thinking on each point:

• Let's instrument this thing in the most standard, basic and low-code way possible. I think all advanced advanced features that require more work from our side can come later as we need them. Let's get something basic rolling first :)

Fully agreed. The plan would be to start with opentelemetry-instrumentation-fastapi for HTTP-level metrics and traces whic is literally one line: FastAPIInstrumentor.instrument_app(app), plus standard OTEL_* env vars for all configuration.

• I see Elastic should already have OpenTelemetry built in

That covers monitoring the ES cluster itself, but we still need to instrument the yente application. One thing worth noting: there's an opentelemetry-instrumentation-elasticsearch package for the ES Python client, but no equivalent for OpenSearch. Since yente supports both backends, I'd propose adding lightweight manual spans at the SearchProvider base class level instead. This keeps observability consistent across both backends and lives in one place.

• The OpenTelemetry Python SDK seems to have a ready-to-go thing for FastAPI.

Yep, that's the plan for the first PR. It gives us request latency, status codes, and trace propagation out of the box. The existing TraceContextMiddleware stays as-is since it serves a separate purpose.

• There seems to be a lot of standard configuration variables. Is there a way we can let our instrumentation be configurable like every other app under the sun with only a single line? Can we make it read the standard OTEL_CONFIG_FILE?

The standard OTEL_* env vars (OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, etc.) are handled automatically by the SDK with zero custom code on our side. OTEL_CONFIG_FILE is a newer spec feature with less ecosystem support, so I'd start with env vars and add file config later if there's demand.

• Adding a /metrics endpoint seems like another standard thing that we would want to have - or not? Maybe the modern way to do this is to only support the OpenTelemetry push paradigm?

I'd support both. OTLP push requires adding the SDK and exporter dependencies but needs no application code, since it's configured entirely via standard OTEL_* env vars. A /metrics Prometheus scrape endpoint would be a small follow-up PR. Many deployments use one or the other, so having both options keeps it flexible.

• Let's keep the PRs small-ish and easily reviewable :)

Thinking roughly:

1. PR 1: FastAPI auto-instrumentation + manual spans on SearchProvider + OTLP export, all via standard env vars
2. PR 2: /metrics Prometheus scrape endpoint
3. PR 3: Custom gauges for catalog data (datasets loaded/outdated, index staleness)

Happy to jump on a call to exchange thoughts as well!