[feat][monitor] Add ML write latency histogram and entry size histogram as OTel metrics

[equanz]

PIP: #21080

Motivation

Please see the PIP-264 doc.

ref.

• https://github.com/apache/pulsar/blob/v4.1.1/pip/pip-264.md#moving-topic-level-histograms-to-namespace-and-broker-level-only
  • Note:

  Histograms are problematic in that sense, since they yet to have an asynchronous version - alas they are only synchronous. Hence, if we use them for topic instruments, used attributes can never be cleared from memory for them thus we’ll have an “attributes” (topic) leak, as over time it will only grow.

  • Therefore, even if a namespace is deleted, the namespace-level histogram metrics persist
• https://github.com/apache/pulsar/blob/v4.1.1/pip/pip-264.md#which-summaries--histograms-are-effected

Modifications

• Added ML write latency histogram and entry size histogram as OTel metrics

Verifying this change

• Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

• Added a simple E2E test to verify the expected metrics is exposed

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
  • Introducing the new OTel metrics
    • pulsar.broker.managed_ledger.message.outgoing.latency
    • pulsar.broker.managed_ledger.message.outgoing.ledger.latency
    • pulsar.broker.managed_ledger.ledger.switch.latency
    • pulsar.broker.managed_ledger.entry.size
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: equanz#14

[equanz]

Note:
In my understanding, implementing AutoConfigurationCustomizerProvider allows you to customize the histogram buckets.

Here is an example.

import com.google.auto.service.AutoService;
import io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizer;
import io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizerProvider;
import io.opentelemetry.sdk.metrics.Aggregation;
import io.opentelemetry.sdk.metrics.InstrumentSelector;
import io.opentelemetry.sdk.metrics.View;
import java.util.Arrays;

@AutoService(AutoConfigurationCustomizerProvider.class)
public class MyCustomizerProvider implements AutoConfigurationCustomizerProvider {

    @Override
    public void customize(AutoConfigurationCustomizer customizer) {
        customizer.addMeterProviderCustomizer(((sdkMeterProviderBuilder, configProperties) -> {
            sdkMeterProviderBuilder.registerView(InstrumentSelector.builder()
                            .setName("pulsar.broker.topic.publish.latency").build(),
                    View.builder()
                            .setAggregation(Aggregation.explicitBucketHistogram(Arrays.asList(0.001, 30.0))).build());
            return sdkMeterProviderBuilder;
        }));
    }
}

$ PULSAR_EXTRA_CLASSPATH='path/to/custom-provider.jar' PULSAR_EXTRA_OPTS=' -Dotel.sdk.disabled=false -Dotel.metrics.exporter=prometheus ' ./bin/pulsar standalone

ref.

• https://opentelemetry.io/docs/languages/java/sdk/#views
• https://opentelemetry.io/docs/languages/java/configuration/#zero-code-sdk-autoconfigure
  • The pulsar-opentelemetry uses AutoConfiguredOpenTelemetrySdk. So, we can use AutoConfigurationCustomizerProvider
    https://github.com/apache/pulsar/blob/v4.1.1/pulsar-opentelemetry/src/main/java/org/apache/pulsar/opentelemetry/OpenTelemetryService.java#L72
• https://opentelemetry.io/ja/docs/languages/java/configuration/#autoconfigurationcustomizerprovider

[lhotari]

LGTM

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.25%. Comparing base (fc65db6) to head (08617a2).
⚠️ Report is 26 commits behind head on master.

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #24815       +/-   ##
=============================================
+ Coverage     36.04%   74.25%   +38.20%     
- Complexity    13055    33403    +20348     
=============================================
  Files          1855     1912       +57     
  Lines        144926   149140     +4214     
  Branches      16814    17300      +486     
=============================================
+ Hits          52243   110741    +58498     
+ Misses        85439    29556    -55883     
- Partials       7244     8843     +1599     

Flag	Coverage Δ
inttests	26.35% <90.19%> (?)
systests	22.73% <100.00%> (-0.04%)	⬇️
unittests	73.77% <100.00%> (+39.28%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...he/bookkeeper/mledger/ManagedLedgerAttributes.java	100.00% <100.00%> (ø)
...kkeeper/mledger/impl/ManagedLedgerFactoryImpl.java	58.86% <ø> (+25.94%)	⬆️
...ookkeeper/mledger/impl/ManagedLedgerMBeanImpl.java	90.38% <100.00%> (+16.73%)	⬆️
.../mledger/impl/OpenTelemetryManagedLedgerStats.java	100.00% <100.00%> (+28.12%)	⬆️

... and 1437 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[equanz]

Rebased to the master to resolve conflicts.