Skip to content

Adds test env delay incident report#7

Open
willik wants to merge 1 commit intomainfrom
test-env-delay-incident-report
Open

Adds test env delay incident report#7
willik wants to merge 1 commit intomainfrom
test-env-delay-incident-report

Conversation

@willik
Copy link
Copy Markdown
Contributor

@willik willik commented Mar 12, 2026

Summary

  • Adds a polished incident report for the ~11-hour test environment delay on the muralmember-lastaccessdate env (mural-api PR #15619)
  • Restructured from an investigation journal into a top-down report with front-loaded takeaways: summary, root cause, recommendations, detailed timeline, and appendices
  • Root cause: force push image cycling created excess pods which, combined with cluster saturation and no autoscaler, left the API pod in FailedScheduling for 9.5 hours

Test plan

  • Review document for factual accuracy
  • Confirm all times are correct in UTC

@willik willik force-pushed the test-env-delay-incident-report branch from 2fef0b4 to de48e92 Compare March 12, 2026 01:45
@willik willik force-pushed the test-env-delay-incident-report branch from de48e92 to b8068a3 Compare March 12, 2026 01:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant