Project: Backpressure Management

[smeubank]

Project Board

See our project board to track this initiative across all SDKs

Description

Problem Statement

In some high throughput situations, specifically, with generation and sending of transaction events, our Backend SDKs may have some negative overhead.

Situationally this can be something like the straw that broke the camels back, meaning the application is already in a bad state, but the SDK should ideally minimize it's impact in the situation.

As a driver, the concern is that we lose metrics data from sampling, so we would like to be able to collect as much data as possible from performance data for high fidelity metrics.

example change which caused this issue:
https://github.com/getsentry/sentry/pull/45592/files

Solution Brainstorm

Internal Vanguard with explanation of imple: here

Develop spec:???

### Tasks
- [ ] https://github.com/getsentry/sentry-docs/issues/7324
- [ ] https://github.com/getsentry/sentry-python/issues/2095
- [ ] https://github.com/getsentry/sentry-php/issues/1563
- [ ] https://github.com/getsentry/sentry-java/issues/2829
- [ ] https://github.com/getsentry/sentry-ruby/issues/2063
- [ ] https://github.com/getsentry/sentry-javascript/issues/8477
- [ ] https://github.com/getsentry/sentry-dotnet/issues/2410
- [ ] Develop Docs
- [ ] Golang

[sl0thentr0py]

this is a rough outline / plan for python

• simulate bad things / very heavy load and try to crash a server
  • only errors
  • errors + perf
  • ignore profiler for now since it has its own complexity
• what does crash mean?
  • out of memory
  • CPU constantly running hot + queue backlog growing unboundedly - this will never clear if rate of growth faster
• go through datastructures where we store most stuff
  • span recorder
  • breadcrumb buffer
  • background worker queue
  • urllib3 poolmanager
• hot code paths
  • profile the SDK itself on a django request
  • serializer
  • frame extraction (try with too many vars)
  • tracemalloc
  • measure average pool.request time
• add limits or simple exponential backoff logic where necessary
  • track stats / client reports when we start dropping stuff?
  • system should recover gracefully from a bad state, make sure this happens somehow (unclear how exactly to implement)
• see what SDK is doing even when everything is off, try to clean up ‘no-op’ scenario
  • we should be as idle as possible
  • turn off random start_spans that are always called

[sl0thentr0py]

related PR where user found UUID generation is slow
getsentry/sentry-python#2131

[sl0thentr0py]

measure background worker queue clearing times

In general, our outbound requests take around 30ms to clear. If the user app runs hot and the queue (capped at 100) is full, it will stay full and drop events till the background worker thread gets roughly 30*100 / 1000 = 3 seconds of CPU time allocated to clear this backlog.

We will record these drops with the queue_overflow Client Report for visibility.

Other than that, I think the background worker is safe because of the queue limit and should not cause stability issues. The queue will never grow in an unbounded manner.

methodology

all following experiments in this thread done on

Software:
    System Software Overview:
      System Version: macOS 13.2.1 (22D68)
      Kernel Version: Darwin 22.3.0
Hardware:
    Hardware Overview:
      Model Name: MacBook Pro
      Model Identifier: MacBookPro16,2
      Processor Name: Quad-Core Intel Core i7
      Processor Speed: 2,3 GHz
      Number of Processors: 1
      Total Number of Cores: 4
      L2 Cache (per Core): 512 KB
      L3 Cache: 8 MB
      Hyper-Threading Technology: Enabled
      Memory: 32 GB
      System Firmware Version: 1916.80.2.0.0 (iBridge: 20.16.3045.0.0,0)
      OS Loader Version: 564.40.4~66

run following with this patch

import sentry_sdk
from threading import Thread
from time import sleep

sentry_sdk.init()

def work():
    for i in range(500):
        try:
            1 / 0
        except Exception:
            sentry_sdk.capture_exception()

def size():
    return sentry_sdk.Hub.current.client.transport._worker._queue.qsize()

def measure():
    while size() > 0:
        sleep(0.1)
        print(f"queue size {size()}")

t1 = Thread(target=work)
t2 = Thread(target=measure)

t1.run()
t2.run()

times = sentry_sdk.Hub.current.client.transport._worker._times
print(f"min: {min(times)}, mean: {sum(times) / len(times)}, max: {max(times)}")

output

min: 0.027270078659057617, mean: 0.030650407617742365, max: 0.0953221321105957

[sl0thentr0py]

• PR which caused s4s to blow up - chore(hybrid-cloud): use sentry performance metrics sentry#44465
• incident link - https://www.notion.so/sentry/INC-305-Large-increase-in-transaction-volume-broke-S4S-63b223db2db54136b67cf43e8b7b7a0d?pvs=4

[mattjohnsonpint]

Any thoughts to priority queue as a pattern here? For example, if we're handling backpressure for errors and transactions in the same queue (or circular buffer, etc.), then a flood of transactions could prevent errors from coming through. Priority queue, or separate queues, would solve that.

[sl0thentr0py]

great suggestion @mattjohnsonpint thanks!
I think separate queues is iffy because it breaks the event agnostic nature of the transport, but we could certainly add weights / priorities when we submit events without changing much of the existing logic.

[mattjohnsonpint]

Some design thought needs to go into this, especially on the background worker. In the .NET SDK, we currently set a default maximum size of the background worker queue to 30 envelopes - and they're not prioritized. In a server environment, a flood of activity can fill up the buffer quickly with transaction envelopes. If an error occurs while the queue is full, the envelope is dropped. At minimum we should probably set a larger default for web SDKs, (ASP.NET Core for .NET, Django in python, etc.). But also we might think about separate limits for different envelope types (ie, 100 transactions, 50 errors, etc.). Either way, this also needs to be documented (In .NET it's MaxQueueItems, which appears absent from the user docs presently.)

Not sure about in Python, but in .NET the background worker operates independently of the transport. The transport itself doesn't manage the queue, but rather the client adds items to the worker's queue, which then uses a transport when dequeuing. Basically we insert a queue managed by a BackgroundWorker between the Client and the Transport seen in the unified API design.

So multiple concurrent queues would be possible as an implementation detail. But perhaps this can vary by SDK.