Skip to content

feat(crons): Send detector_id along with the occurrence, when available #97963

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

wedamija
Copy link
Member

This starts sending the detector_id to the issue platform, so that we can trigger workflows.

…able

This starts sending the `detector_id` to the issue platform, so that we can trigger workflows.
@wedamija wedamija requested a review from a team August 15, 2025 22:53
@wedamija wedamija requested a review from a team as a code owner August 15, 2025 22:53
@github-actions github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Aug 15, 2025
@@ -144,6 +145,11 @@ def send_incident_occurrence(
if last_successful_checkin:
last_successful_checkin_timestamp = last_successful_checkin.date_added.isoformat()

detector = get_detector_for_monitor(monitor_env.monitor)
evidence_data = {}
if detector:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if I should also put this behind a feature flag, but I think it's not likely to cause many issues. What do you think @saponifi3d?

Copy link
Contributor

@saponifi3d saponifi3d Aug 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right now this should be fine because the post_process is only setup for error events and metric issues -- so this should be good 👍

we will need issue alerts to start responding to more than just error events soon -- but we'll add an allow/deny list when we go to do that so it's easy to enable/disable different issue types.

Comment on lines +148 to +151
detector = get_detector_for_monitor(monitor_env.monitor)
evidence_data = {}
if detector:
evidence_data["detector_id"] = detector.id
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential bug: The call to get_detector_for_monitor only handles DoesNotExist errors. Other database exceptions will crash send_incident_occurrence, preventing incident creation.
  • Description: The get_detector_for_monitor function is called to retrieve an optional Detector. This function's database query only handles the Detector.DoesNotExist exception. Other potential database exceptions, such as OperationalError from a connection timeout or transaction issue, are not caught. When send_incident_occurrence calls this function, an unhandled database exception will propagate and cause the caller to crash. As a consequence, the system will fail to create the incident occurrence, silently dropping a critical monitoring alert.

  • Suggested fix: Update get_detector_for_monitor to catch broader database exceptions, such as OperationalError and DatabaseError, alongside DoesNotExist. In these cases, the function should return None. This prevents transient database issues from crashing the send_incident_occurrence process and ensures incident occurrences can still be created, even if the optional detector information cannot be retrieved.
    severity: 0.65, confidence: 0.95

Did we get this right? 👍 / 👎 to inform future reviews.

Copy link

codecov bot commented Aug 15, 2025

❌ 2 Tests Failed:

Tests completed Failed Passed Skipped
24785 2 24783 542
View the top 2 failed test(s) by shortest run time
tests.sentry.monitors.consumers.test_monitor_consumer.MonitorConsumerTest::test_failing
Stack Traces | 2.7s run time
#x1B[1m#x1B[.../monitors/consumers/test_monitor_consumer.py#x1B[0m:282: in test_failing
    checkin = MonitorCheckIn.objects.get(guid=self.guid)
#x1B[1m#x1B[31m.venv/lib/python3.13.../db/models/manager.py#x1B[0m:87: in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
#x1B[1m#x1B[31m.venv/lib/python3.13.../db/models/query.py#x1B[0m:633: in get
    raise self.model.DoesNotExist(
#x1B[1m#x1B[31mE   sentry.monitors.models.MonitorCheckIn.DoesNotExist: MonitorCheckIn matching query does not exist.#x1B[0m
tests.sentry.monitors.consumers.test_monitor_consumer.MonitorConsumerTest::test_check_in_update_terminal_user_terminal
Stack Traces | 2.79s run time
#x1B[1m#x1B[.../monitors/consumers/test_monitor_consumer.py#x1B[0m:524: in test_check_in_update_terminal_user_terminal
    self.send_checkin(
#x1B[1m#x1B[.../monitors/consumers/test_monitor_consumer.py#x1B[0m:111: in send_checkin
    with self.check_processing_errors(wrapper, expected_error, expected_monitor_slug):
#x1B[1m#x1B[.../hostedtoolcache/Python/3.13.1.../x64/lib/python3.13/contextlib.py#x1B[0m:148: in __exit__
    next(self.gen)
#x1B[1m#x1B[.../monitors/consumers/test_monitor_consumer.py#x1B[0m:144: in check_processing_errors
    assert len(args_list) == 1
#x1B[1m#x1B[31mE   assert 0 == 1#x1B[0m
#x1B[1m#x1B[31mE    +  where 0 = len([])#x1B[0m

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Scope: Backend Automatically applied to PRs that change backend components
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants