Performance issues after updating to Kafka 4

[AnnChristinK]

Hello everyone!
After updating our Kafka Connect base image to Kafka 4 we came across the issue that our perfomance got really bad.

After looking into it we found out that our connector is only able to produce one event every 20 seconds. As of right now it is unclear if the delay is while reading from MQ or writing into our kafka topic.

Do you have any hints what we could configure to make it faster again? Our current attempts to not seem to work.

Is this issue known? Is Kafka 4 officially supported already or do we need to wait for one of the next releases?

Thank you for your answers and have a wonderful day!
Ann-Christin

[Joel-hanson]

Thank you for raising the issue

We haven't done any performance comparison on Kafka 4 yet, but we have noticed a couple of deprecated methods, which we are actively working to fix: #159. Would you mind sharing the connector logs so we can investigate this further?

Also, could you please share the version of the MQ source connector you're using? And how you are setting up your connector. If you are using the latest MQ source connector we recently added a few configurations to control the overall polling mechanism, such as mq.receive.subsequent.timeout.ms and mq.message.receive.timeout, Have you tried changing those?

[KBGenerali]

Hello @Joel-hanson,
I'm a colleague of @AnnChristinK and looked further into this. I made a test on our test environment to share the logs with you (see log.txt). I tried to strip down the log to all the necessary information.
In the log you can see that every 10 seconds there is a processing of a queue message but only every second fetch finds a message to process.

We use the MQ source connector in the latest version (2.5.0) but we didn't look into the new configurations. We don't have these problems if we use kafka 3.9.1 as our base image.

Our overall setup

• We use Docker to build an image that runs as a kafka connect cluster container in OpenShift
• The base image is apache/kafka:3.9.1 > after the update apache/kafka:4.0.0
• We download the plugin jar from github-releases
• We install the plugin via connect-plugin-path.sh script
• The connectors are configured via the REST API and run with tasks.max = 1, mq.batch.size = 1, idempotence for producer is enabled
• We have multiple connectors configured that listen to different queues
• We run the connect cluster in single mode
• Our queues run on IBM MQ z/OS