Tracing uses a lot of memory

[ebeigarts]

I'm using nest.js with sentry (9.46.0) and bullmq.

I have wrapped each job execution with Sentry.withIsolationScope:

// https://docs.sentry.io/platforms/javascript/guides/nestjs/tracing/instrumentation/queues-module/
// https://docs.sentry.io/platforms/javascript/guides/node/enriching-events/request-isolation/
const withSentrySpan = async (
  job: Job<any, any, string>,
  queueProcessor: string,
  callback: (job: Job<any, any, string>) => Promise<void>,
): Promise<void> => {
  return Sentry.withIsolationScope(() =>
    Sentry.startSpan(
      {
        name: job.name,
        op: queueProcessor,
        forceTransaction: true,
        attributes: {
          id: job.id,
          name: job.name,
          data: JSON.stringify(job.data),
        },
      },
      async (span) => {
        await Sentry.startSpan(
          {
            name: job.name,
            op: 'queue.process',
            attributes: {
              'messaging.message.id': job.id,
              'messaging.destination.name': job.queueQualifiedName,
              'messaging.message.body.size': 0,
              'messaging.message.receive.latency': Date.now() - job.timestamp,
              'messaging.message.retry.count': job.attemptsMade,
            },
          },
          async () => {
            try {
              const res = await callback(job)
              span.setStatus({ code: 1 }) // OK
              return res
            } catch (error) {
              span.setStatus({ code: 2 }) // ERROR
              throw error
            }
          },
        )
      },
    ),
  )
}


@Processor('default', {
  concurrency: 8,
  stalledInterval: 240000,
  maxStalledCount: 1,
  lockDuration: 240000,
  lockRenewTime: 120000,
})
export class DefaultProcessor extends BaseProcessor {
  @CreateRequestContext()
  async process(job: Job<any, any, string>): Promise<void> {
    return withSentrySpan(job, 'queue.default_processor', async () => {
      ...
    })
  }
}

But still my background worker is using a lot of memory and it seems to be growing.
The memory keeps growing even after there are no jobs active and the queue is empty.
Somehow the spans are still left in the memory.

If I disable tracing tracesSampleRate: 0.0 then memory usage is OK.

Each job execution can take up to 15 minutes by doing a lot of HTTP calls, LLM calls etc.

1. Is there a way to limit Span size in Sentry?
2. Can I clear the spans manually after the job has been completed?

Any ideas how can I avoid the memory issues without disabling tracing?

[ebeigarts]

Is it possible to discard span children? I still want to track queue.process spans to be able to see some stats in sentry, but for me it is fine if I could just drop the nested spans.

[mydea]

hey, do you mind opening an issue with this, if memory is being leaked (which it appears to do in your example) this is a bug that we def. want to fix!
With long running spans it may take longer to clean up but it should eventually clean up memory even then 🤔

You could try to not use startSpan to wrap this but manually wrap it with an inactive span, then child spans will disregard it - not sure if this will help you, but you can try:

 return Sentry.withIsolationScope(() => {
  const span = Sentry.startInactiveSpan(...);
            try {
              const res = await callback(job);
              span.setStatus({ code: 1 }) // OK
              return res
            } catch (error) {
              span.setStatus({ code: 2 }) // ERROR
              throw error
            } finally {
              span.end();
          }
})