Sentry + Opentelemetry exports at the same time

[nathanscully]

I am in the process of migration Sentry 7 to 8. We already have an existing Open telemetry integration that is pushing traces and metrics to a otel collector, which we are consuming in grafana.
I have managed to migrate this to using the default sentry.init(..) function and can see traces appearing in my sentry console. However I expected having a locally running collector to also receive data. I'd still like to push the otel data to the collector as well as directly to Sentry. It wasn't clear from the docs (that I could find) how to have multiple transports to achieve this?

Any advice?

[AbhiPrasad]

@nathanscully please see: https://docs.sentry.io/platforms/javascript/guides/node/tracing/instrumentation/opentelemetry/#using-a-custom-opentelemetry-setup

[mydea]

FYI we have shipped new docs for this which should make this clearer, hopefully: https://docs.sentry.io/platforms/javascript/guides/node/opentelemetry/custom-setup/ or https://docs.sentry.io/platforms/javascript/guides/node/opentelemetry/using-opentelemetry-apis/#modifying-the-default-opentelemetry-tracerprovider if you only want to send the trace data to a different backend in addition to Sentry.

[sbriceland]

@AbhiPrasad and @mydea - The documentation does not help with the problem mentioned. SentrySampler "rules them all". When span processors are added they still adhere to the behavior of the sampler.

[mydea]

On which version of the Sentry SDK are you? In the latest version, stuff should work, in theory at least.
The SentryContextManager is def. needed, without it you cannot get request isolation which will break a lot of core Sentry functionality.

The sampler and propagator you may skip, but then you will not get proper trace propagation between sentry services. If you do not care about this, stuff should still work, mostly.

We have tests covering this scenario here: https://github.com/getsentry/sentry-javascript/blob/develop/dev-packages/e2e-tests/test-applications/node-otel-without-tracing/src/instrument.ts
Note there is a big difference between enableTracing: false and not configuring this at all - this is a bit unfortunate and will change in v9, but for now, you need to actually omit both enableTracing and tracesSampleRate completely from your config to get your desired outcome. Possibly this is the problem you are facing there.

[dvictory]

Thanks, I did get my intended outcome by excluding tracesSampleRate but setting enableTracing to true. But I have a second question. Is it possible to use the SentrySpanProcessor without using the SentrySampler? I want my application to emit 100% of the spans I have instrumented because we have tail sampling in use which handles the reduction of data (which is what tracesSampleRate along with the SentrySampler do for sentry). If I use the SentrySampler and I don't have the rate set to 1, I lose spans. Ideally tracesSampleRate for sentry would be respected, but ignored for the OtelExporter which would send 100% of spans out. For now I am not using the SentrySampler, does this mean that I will get 100% of spans to sentry regardless of the value of tracesSampleRate? or does it just mean that propagation will have issues because the sampler decision is not propagated, but I will still only get a subset of spans?

what's coming in v9? What about dedicated ingest endpoints for Otel spans (like other platforms) where I can just point my collector to sentry and not worry about exporting spans to Sentry in my app?

Thanks

[mydea]

So as of now there is not a built-in way to send some spans to sentry, but all spans to other exporters - you'll have to set tracesSampleRate: 1 to emit all spans, but you can add a beforeSendTransaction(event) hook and conditionally drop whole transactions there (by returning null from the callback) to be sent to sentry. This should lead to all spans being sent to other exporters, but only some/less being send to sentry. This is with using the SentrySampler. You should not leave out the SentrySampler, because it is needed to ensure trace propagation works even for unsampled traces (so errors can still be correlated etc).

Regarding v9, not a whole lot is planned in this regard. We are looking into dedicated OTLP endpoints but this is very much WIP without a clear due date!

[ryanjdillon]

@mydea Are there any updates regarding the OTLP endpoints?

[AbhiPrasad]

@ryanjdillon please see getsentry/sentry#85902.

There are no updates on the timeline atm, but we are hard at work making this happen!