PIP-211 [feat][tiered-storage] Introduce offload throttling, Part 1#17485
PIP-211 [feat][tiered-storage] Introduce offload throttling, Part 1#17485tjiuming wants to merge 18 commits intoapache:masterfrom
Conversation
|
@hangc0276 @codelipenghui PTAL |
|
I'll create another PR to support restApi/cmd to modify offload policies after the PR merged |
|
Tests to be completed. |
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
Outdated
Show resolved
Hide resolved
# Conflicts: # managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
|
@tjiuming, Could you please create a proposal for this one? We will introduce a new mechanism, we need to consider how to operate and monitor the new feature. |
@codelipenghui Yes, I'm re-designing the flow-controller, the current implementation will block read thread and will take effect to other ledger's reading. After that, I'll open a proposal |
|
The pr had no activity for 30 days, mark with Stale label. |
# Conflicts: # managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerConfig.java # managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java # pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
|
Since we will start the RC version of
So drag this PR to |
|
The pr had no activity for 30 days, mark with Stale label. |
|
Replaced by #22385 |
Fixes #17362
Motivation
Support offload throttling, to prevent offloading use up all the broker resources(CPU, network).
From the perspective of generality, we can't limit how many bytes write to tiered-storage, it will invade the corresponding implementations(filesystem, streaming, etc.). So we can only to limit how many bytes read from ML, this cannot absolutely limit the traffic per second.
Modifications
TimeWindow.java、WindowWrap.java、TimeWindowTest.javafrombrokermodule tomanaged-ledgermodule.managedLedgerGlobalOffloadingPermitBytesPerSecondconfiguration, it works on the broker level.globalOffloadingPermitBytesPerSecondintoManagedLedgerConfig.Does this pull request potentially affect one of the following parts:
If
yeswas chosen, please highlight the changesDocumentation
Check the box below or label this PR directly.
Need to update docs?
doc-required(Your PR needs to update docs and you will update later)
doc-not-needed(Please explain why)
doc(Your PR contains doc changes)
doc-complete(Docs have been already added)
Matching PR in forked repository
PR in forked repository: tjiuming#11