feat: always-on TrackedMutex for lock contention instrumentation#344
Merged
feat: always-on TrackedMutex for lock contention instrumentation#344
Conversation
comstud
reviewed
Mar 18, 2026
comstud
reviewed
Mar 18, 2026
comstud
reviewed
Mar 18, 2026
3ff3b37 to
3e07841
Compare
Replace sync.Mutex with TrackedMutex on all entity structs to detect deadlocks and lock contention at runtime. Fast path uses TryLock with ~25ns overhead. On contention, logs holder/waiter identity and wait duration. Warns if any lock is held longer than 5 seconds. All get*Record* functions now accept a caller string parameter that propagates through to Lock(), enabling precise identification of which code path holds or is waiting for a lock in contention logs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… IDs - TrackedMutex[K] is now generic over the entity ID type; string formatting (%v) only runs on the cold contention/warning paths - holder and acquiredAt use atomic.Value / atomic.Int64 so reads from the contention path are race-detector safe - Replace Init+sync.Once with direct parameters on Lock/Unlock for simplicity and smaller struct size - Consolidate repeated time.Now() calls into local variables - Skip incidents with blank IncidentId in UpdateFortBatch to avoid cache key collisions and spurious contention Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add 1-2-4-8-16-32-64-128ms backoff loop (~255ms total) before emitting LOCK_CONTENTION warnings. Most transient contention resolves silently; only locks held longer than 255ms produce log output. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
618897d to
dc4e92e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TrackedMutex(decoder/tracked_mutex.go) wrappingsync.Mutexwith TryLock-based contention detection and caller trackingsync.MutexwithTrackedMutexon all 9 entity structs (Pokestop, Gym, Incident, Pokemon, Station, Weather, Spawnpoint, Route, Tappable)caller stringparameter to all 31get*Record*/Peek*functions, threaded through toLock(caller)How it works
TryLock()succeeds, stores caller + timestamp — zero log overhead[LOCK_CONTENTION]with entity type/id, waiter, holder, and hold duration, then blocks. Logs[LOCK_ACQUIRED]once acquired with wait time[LOCK_HELD_LONG]on Unlock if held >5sExample output
Test plan
go build ./...compiles cleanlygo vet ./...passesgo test ./...all tests pass[LOCK_CONTENTION]appears only during actual contention, silence otherwise🤖 Generated with Claude Code