Skip to content

Circuit breaker for Google Workspace comment poller (#1490)#1494

Open
bingran-you wants to merge 2 commits intodevfrom
bry/fix-gworkspace-circuit-breaker-1490
Open

Circuit breaker for Google Workspace comment poller (#1490)#1494
bingran-you wants to merge 2 commits intodevfrom
bry/fix-gworkspace-circuit-breaker-1490

Conversation

@bingran-you
Copy link
Copy Markdown
Contributor

Summary

  • Google's comments API returns sticky 5xx for some files (staging example: a presentation that permanently returns `500 internalError`). The poller ran `list_comments` every cycle, producing unbounded log spam and wasting Drive API quota.
  • Added `CommentCircuitBreaker` keyed on `file_id`. After 5 consecutive failures the file is skipped with exponential cooldown (1m, 2m, 4m ... capped at 30m). Success immediately resets. Warn emitted at trip time, then at most once per hour.
  • Wired into all three pollers (Docs, Sheets, Slides) — they share the same failure mode.

Refs #1490

Test plan

  • `cargo test -p scheduler_module --lib google_workspace_poller` — 5 new tests pass (threshold, trip, reset, isolation, max cooldown cap)
  • `cargo check -p scheduler_module` clean (only pre-existing warnings)
  • Verify on staging: gateway no longer emits per-cycle `500 internalError` spam for presentation `1zsQXUFU9sbVLWCojAI0si5Hh_BxS1HvGbwyFO_yv_3w`

Google's comments API occasionally returns sticky 5xx errors for
specific files (observed on a staging presentation: permanent 500
internalError). The poller runs list_comments every cycle, so those
files produced continuous log spam and wasted API quota.

Introduce CommentCircuitBreaker keyed on file_id:
- After 5 consecutive list_comments failures, skip that file for an
  exponentially growing cooldown window (1m, 2m, 4m ... capped at 30m).
- Elevated warn emitted at trip time and then at most once per hour
  while the breaker stays open.
- Any success immediately resets state.

Applied to all three pollers (Docs, Sheets, Slides) since they share
the same failure mode.

Refs #1490
@vercel
Copy link
Copy Markdown

vercel Bot commented Apr 21, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
dowhiz Ready Ready Preview, Comment Apr 21, 2026 8:08pm

@bingran-you
Copy link
Copy Markdown
Contributor Author

CI rust check was failing on cargo fmt --all --check. Pushed 09f18cf3 applying cargo fmt --all — only whitespace/line-wrap changes in google_workspace_poller.rs (no logic change). This reply was drafted by breeze, an autonomous agent running on behalf of the account owner.

@bingran-you bingran-you added breeze:done Breeze finished handling this item and removed breeze:wip Breeze is actively working on this item labels Apr 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breeze:done Breeze finished handling this item

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant