[fix][client] Make auto partitions update work for old brokers without PIP-344

[BewareMyPower]

Motivation

See https://lists.apache.org/thread/tpd2kgl4v3x8bkw0skv0nj38rzrk56t7

Modifications

Set the 3rd argument of getPartitionedTopicMetadata to true in PulsarClientImpl#getPartitionsForTopic, which is used to query the number of partitions.

Add testAutoPartitionsUpdate to cover the case. Specially, it tests a consumer that subscribes two partitioned topics to cover the path here:

pulsar/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java

Line 182 in 008e4eb

.map(t -> subscribeAsync(t, createTopicIfDoesNotExist))

P.S. This test is only added to ClientTest25, which runs tests against apachepulsar/pulsar:2.5.0 image. The current backward compatibility test strategy is ambiguous that only 2.2.0, 2.3.0, 2.4.0, 2.5.0 are tested. It's hard to cover all old versions, but only choose these 4 versions does not make sense. This topic should be out of the scope of this PR. Anyway, the test on 2.5.0 can already prevent regression on the fix.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

[lhotari]

LGTM, thanks for addressing this

[lhotari]

There seems to be a test that explicitly tests for the behavior that connecting to an older broker fails:

  Error:  Tests run: 6, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 32.05 s <<< FAILURE! -- in org.apache.pulsar.client.impl.ClientCnxTest
  Error:  org.apache.pulsar.client.impl.ClientCnxTest.testSupportsGetPartitionedMetadataWithoutAutoCreation -- Time elapsed: 0.255 s <<< FAILURE!
  java.lang.AssertionError: Expected an error that the broker version is too old.
  	at org.testng.Assert.fail(Assert.java:110)
  	at org.apache.pulsar.client.impl.ClientCnxTest.testSupportsGetPartitionedMetadataWithoutAutoCreation(ClientCnxTest.java:264)
  	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
  	at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47)
  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76)
  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  	at java.base/java.lang.Thread.run(Thread.java:1583)

This test was added in #22773 and it seems to test what is defined in PIP-344 Backward and Forward compatibility:

• New version client and Old version Broker: The feature flag supports_binary_api_get_partitioned_meta_with_param_created_false will be false. The client will get a not-support error if the param createIfAutoCreationEnabled is false.

@poorbarcode would it be possible to explain the reason why the client should fail in this case?

[BewareMyPower]

I think it has been explained in #23136. I updated the test and the PIP document now.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.29%. Comparing base (008e4eb) to head (6086925).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #24822      +/-   ##
============================================
- Coverage     74.42%   74.29%   -0.13%     
- Complexity    33488    33801     +313     
============================================
  Files          1912     1912              
  Lines        149077   149089      +12     
  Branches      17300    17300              
============================================
- Hits         110950   110769     -181     
- Misses        29331    29482     +151     
- Partials       8796     8838      +42     

Flag	Coverage Δ
inttests	26.38% <0.00%> (-0.33%)	⬇️
systests	22.71% <0.00%> (-0.18%)	⬇️
unittests	73.83% <100.00%> (-0.11%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...rg/apache/pulsar/client/impl/PulsarClientImpl.java	76.11% <100.00%> (ø)

... and 91 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.