Skip to content

[improve] add read rate limter for compaction#8

Open
HQebupt wants to merge 1 commit intomasterfrom
addReadRateLimterInCompaction
Open

[improve] add read rate limter for compaction#8
HQebupt wants to merge 1 commit intomasterfrom
addReadRateLimterInCompaction

Conversation

@HQebupt
Copy link
Owner

@HQebupt HQebupt commented May 31, 2023

Motivation

The purpose of this feature is to provide more flexibility and control over the compaction process in the Bookkeeper server by allowing users to configure the read rate of compaction according to their specific requirements.
It adds support for configuring the read rate of compaction in bytes per second.

In our production environment of Bookkeeper, we have set compactionRateByBytes=30M. However, we have observed that it only limits the write speed to disk and does not limit the read speed from disk. In our environment, the continuous saturation of disk IO for reads in compaction is causing issues with data consumption from the disk.

// Set the rate at which compaction will readd entries. The unit is bytes added per second.
 compactionRateByBytes=31457280

image

image

Changes

  • In the GarbageCollectorThread class, a new field compactionReadByteRateLimiter of type RateLimiter is added to control the read rate of compaction.
  • In the ServerConfiguration class, a new configuration property COMPACTION_READ_RATE_BY_BYTES is defined to specify the read rate of compaction in bytes per second.

Master Issue: #8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant