[feat][monitor] Add publish latency histogram as OTel metrics

[equanz]

PIP: #21080

Motivation

Please see the PIP-264 doc.

ref.

• https://github.com/apache/pulsar/blob/v4.1.1/pip/pip-264.md#switching-summary-to-histograms-in-namespacebroker-level-only
• https://github.com/apache/pulsar/blob/v4.1.1/pip/pip-264.md#which-summaries--histograms-are-effected

Modifications

• Added publish latency histogram as OTel metrics
• Added PulsarDeprecatedMetric annotation to the pulsar_broker_publish_latency Prometheus metrics

Verifying this change

• Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

• Added a simple E2E test to verify the expected metrics is exposed

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
  • Introducing the new OTel metrics pulsar.broker.topic.publish.latency
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: equanz#13

[lhotari]

LGTM

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.25%. Comparing base (e44e084) to head (498fe16).
⚠️ Report is 5 commits behind head on master.

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #24810       +/-   ##
=============================================
+ Coverage     38.36%   74.25%   +35.89%     
- Complexity    13171    33763    +20592     
=============================================
  Files          1854     1912       +58     
  Lines        144870   149084     +4214     
  Branches      16808    17299      +491     
=============================================
+ Hits          55574   110704    +55130     
+ Misses        81752    29541    -52211     
- Partials       7544     8839     +1295     

Flag	Coverage Δ
inttests	26.25% <100.00%> (+0.02%)	⬆️
systests	22.68% <100.00%> (+<0.01%)	⬆️
unittests	73.77% <100.00%> (+39.14%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...rg/apache/pulsar/broker/service/AbstractTopic.java	88.35% <100.00%> (+22.11%)	⬆️
.../org/apache/pulsar/broker/service/PulsarStats.java	87.02% <100.00%> (+3.30%)	⬆️
.../pulsar/broker/stats/BrokerOperabilityMetrics.java	94.69% <100.00%> (+1.07%)	⬆️

... and 1412 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.