-
Notifications
You must be signed in to change notification settings - Fork 74
[flags] fix: use write lock for addListener #3132
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: develop
Are you sure you want to change the base?
Conversation
| val listener1 = object : FlagsStateListener { | ||
| override fun onStateChanged(newState: FlagsClientState) { | ||
| if (newState is FlagsClientState.Ready) { | ||
| if (newState == FlagsClientState.Ready) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FlagClientState is a sealed class
|
|
||
| @Test | ||
| fun `M stop notifying subsequent listeners W updateState() { if listener throws }`() { | ||
| fun `M stop notifying subsequent listeners W updateState() { listener throws }`() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
polishing some names/comments while here
|
🎯 Code Coverage 🔗 Commit SHA: 0558c4c | Docs | Datadog PR Page | Was this helpful? Give us feedback! |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #3132 +/- ##
========================================
Coverage 70.73% 70.73%
========================================
Files 893 893
Lines 33000 33000
Branches 5549 5550 +1
========================================
+ Hits 23341 23342 +1
+ Misses 8102 8097 -5
- Partials 1557 1561 +4
🚀 New features to boost your workflow:
|
| @Test | ||
| fun `M block getCurrentState calls W addListener() { slow listener notification }`() { | ||
| // Given | ||
| val stateOld = FlagsClientState.NotReady |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: probably should be oldState to be consistent with naming convention used in newState.
Or maybe for the clarity we can just inline FlagsClientState.NotReady where needed, because stateOld is val anyway and cannot be changed.
| addListenerThread.start() | ||
| addListenerStarted.await() | ||
| getCurrentStateThread.start() | ||
| getCurrentStateAttempted.await() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
getCurrentStateAttempted is not actually needed I think, the necessary wait for the getCurrentStateThread completion is already implemented by the getCurrentStateThread.join() call below.
| val addListenerSlowCallbackStarted = CountDownLatch(1) | ||
| val getCurrentStateAttempted = CountDownLatch(1) | ||
|
|
||
| val operationTimestamps = mutableListOf<Pair<String, Long>>() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
synchronized calls can be removed if type here is CopyOnWriteArrayList
| } | ||
|
|
||
| @Test | ||
| fun `M block getCurrentState calls W addListener() { slow listener notification }`() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we want to block getCurrentState when addListener is called, if addListener is not changing the state?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🤔 We may have had the right lock here in the first place...
If addListener holds the read lock, then, ostensibly, nothing can write to the state (current state or the listeners). While addListener is technically mutating the protected state by adding a listener, the underlying mechanism itself is theadsafe/synchronized via CopyOnWriteArrayList so it's fine if parallel calls to addListener attempt to mutate the set of listeners. The order of listener subscription is not important to maintain here.
We should just be able to rely on the underlying thread-safety of CopyOnWriteArrayList to avoid clobbering of the listener list, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we shouldn't wrap adding listener with lock then?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need to block writing to the current state until the first notification and addListener are complete. We don't get that atomicity with DDCoreSubscription, only that the calls underlying calls to add the listener to the list are thread safe and synchronized so no listener will be lost when parallel calls are made.
In effect, I believe we can drop this change altogether and rely on the read lock in conjunction with the underlying lock in CopyOnWriteArrayList
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think yes, we should. My thinking is the following:
getCurrentStatecovers onlyFlagsStateManager.currentState. Listeners is not a part ofFlagsStateManager.currentState, so they are not related to thegetCurrentStatecall.FlagsStateManager.addListenercall doesn't modifyFlagsStateManager.currentState, it only reads. So that us why we added a read lock. If something is in the process of modifyingFlagsStateManager.currentState, then there will be a wait to acquirereadlock.FlagsStateManager.addListenermutates underlying listeners collection, but anyway this is atomic and thread-safe,currentStatevalue passed down is guarded by the read lock and if there is iteration already happening inupdateState -> subscription.notifyListeners, then this new listener won't be a part of ongoing iteration (due to the usage ofCopyOnWriteArrayListinDDSubscription).
So I'm curious why should we block getCurrentState call during FlagsStateManager.addListener invocation.
The race condition could otherwise result in a listener missing a state change
Is it about FlagsStateManager.currentState property here? From the code I see, listener shouldn't miss any state change, since read lock cannot be acquired if write lock is active, and the opposite.
What does this PR do?
Uses a
writelock instead ofreadwhen adding a listenerMotivation
Adding a listener effectively changes the contested state we want to keep atomic {
currentState,listeners} so it must block otherwritecalls. The race condition could otherwise result in a listener missing a state changeAdditional Notes
Anything else we should know when reviewing?
Review checklist (to be filled by reviewers)