[fix][broker]If there is a deadlock in the service, the probe should return a failure because the service may be unavailable #10
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
15 tasks
…pache#24313) ### Motivation There is still a call to the synchronous method in `internalCreatePartitionedTopic`, which can lead to a deadlock under certain conditions. ### Modifications - Replaced synchronous call to `getNamespacePolicies()` with `getNamespacePoliciesAsync()`. - Replaced synchronous call to `getTopicPartitionList()` with `getTopicPartitionListAsync()`. - Updated `internalCreatePartitionedTopic` to propagate the async flow properly.
### Motivation we can now obtain the offset of a message by its message id: 1. Get the message by id using `get-message-by-id` cmd 2. Get the index of the message using `Message.getIndex()` But we cannot obtain the message id by offset. Then we need to add a new API to get the message id by offset. ### Modifications Add a new http API to retrieve the message ID by offset. We propose to add a new API to retrieve the message ID by offset, enabling us to cache the mapping between message ID and offset. This will allow us to use offsets for seek and acknowledgment operations when consuming messages through the standardized API.
…picWithRollbackDuration (apache#24318) ### Motivation  The root cause of this problem is that the `reader` is not cleaned up after the unit test `shouldSupportCancellingReadNextAsync` is executed. ### Modifications Add @cleanup on `reader` in test `shouldSupportCancellingReadNextAsync`.
…ltipleTimesWhenTrimLedgers (apache#24330)
…bStoreBackedReadHandleImplV2.readAsync (apache#24331)
…tion deletion rate (apache#24190) Co-authored-by: zjxxzjwang <zjxxzjwang@tencent.com>
Co-authored-by: Lari Hotari <lhotari@users.noreply.github.com>
…s to avoid duplicate Reads (apache#24346)
…ed state (apache#24352) Signed-off-by: Zixuan Liu <nodeces@gmail.com>
…opic deleted, even if the partitioned topic still exists (apache#24350)
…ry (apache#19783) Co-authored-by: Lari Hotari <lhotari@apache.org>
…nt message dispatchers (apache#24386)
…pache#24658) Co-authored-by: Zixuan Liu <nodeces@gmail.com>
…ER_WATER_MARK to pulsar conf and pause receive requests when channel is unwritable (apache#24510)
…ers and printed noisy error logs (apache#24634)
…her than FastThreadLocalThread (apache#24719)
…ng the parsed instance in the broker cache (apache#24682)
… Delivery (apache#24625) Co-authored-by: Christina <qwang3@paypal.com>
…tadataStateStoreProviderImpl.init` (apache#24721)
…received a large response from ZK (apache#24580)
…ng different threads (apache#24725)
… from the replay queue after a consumer disconnects and leaves a backlog (apache#24736) Co-authored-by: Nikolai Borisov <nikolai.borisov@onde.app>
… lost when concurrently occupying topic owner (apache#24722)
…s of retention policy (apache#24733) Co-authored-by: Jiwe Guo <technoboy@apache.org>
Co-authored-by: oneby-wang <onebywang@qq.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…return a failure because the service may be unavailable
Fixes #xyz
Main Issue: #xyz
PIP: #xyz
Motivation
In some special scenarios, when the broker service has a deadlock, it needs to be able to automatically recover instead of requiring manual intervention. For example, when the service is deployed in a customer environment, we cannot directly manage it. If the service has a deadlock, the probe should return a failure because the service may be unavailable. The probe failure triggers a node restart to resolve the deadlock.
Modifications
Add deadlock detection in the probe. If a deadlock exists, print the thread stack and return a service unavailable exception
Verifying this change
(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)
Does this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes
Documentation
docdoc-requireddoc-not-neededdoc-completeMatching PR in forked repository
PR in forked repository: