deployment: do not deploy SigNoz locally

[ret2libc]

Local SigNoz/Langfuse use a lot of resources that make local testing much more resource heavy (e.g. clickhouse is quite big). Instead, just assume users who wants this will deploy these resources separately elsewhere.

This also has the advantage that each CRS deployment/un-deployment does not recreate the observability tools as well. The resources in minikube will be only for the real CRS functionalities and deploying and un-deploying the system will be faster.

[hbrodin]

Looks good to me. I think this is the right thing todo.

[michaelbrownuc]

I think this makes sense, but individual users need these telemetry streams to understand what the CRS is doing in a user friendly way. This is a big upgrade over watching logs in k8s. I think the better option is to just make the local deployments configurable, with the default being that they are deployed. Users who have their own instances can then disable the local deployments and optionally reconfigure with their own URL etc.

[ret2libc]

Without this PR signoz is already deployed locally by default, but I'd say that was not the best choice as signoz and similar are quite heavy and the way we deploy it is tied to the CRS lifecycle. So every time you re-deploy the CRS, you also re-deploy Signoz, which makes things slow, heavy, and it requires more resources in minikube deployments (also, you have to re-configure the user account when you access Signoz and delete browser cookies).

As the local Signoz deployment is mainly for testing, maybe we could have some other ways that people can easily use without requiring them to deploy it. I'm looking right now at Grafana Cloud: it's cloud based, so you don't need to deploy anything, you can just use it for free (with limits), and it should integrate the same way as Signoz with OpenTelemetry. With that, people who wants to see the logs and traces would have to register an account there (or similar services) and then provide the link/token.

WDYT?

[michaelbrownuc]

I don't think we should add an external dependency for a cloud telemetry aggregator.

But I think a good compromise might be to make the local langfuse / signoz disabled by default, but ask the user during configuration if they want advanced telemetry / introspection features. This would let users who want something more than the web GUI turn these features on at higher cost. Does this work?