Frequent Galera Health check timeouts

[ThomasVerhoeven1998]

Issue Summary

ProxySQL intermittently marks healthy Galera nodes as OFFLINE due to health check timeouts. This behavior is most consistent immediately after ProxySQL startup, where we frequently see >5 timeouts within the first few seconds. And during traffic this keeps on happening a lot more than we expect.

The mysql_server_ping_log and mysql_server_connect_log confirm the network and MySQL listeners are highly responsive (latencies in microseconds). However, the mysql_server_galera_log shows success times spiking into hundreds of milliseconds or failing entirely. This, combined with the Watchdog error, points to internal thread starvation rather than actual backend database issues.

Observed Errors

2026-04-15 04:00:46 MySQL_Monitor.cpp:2633:monitor_galera_thread(): [ERROR] Got error. mmsd 0x7fd94c210200 , MYSQL 0x7fd94b3f2d00 , FD 75 : timeout check

Server sql01:3306 missed 3 Galera checks. Assuming offline

Timeout on Galera health check for svevert8sql01:3306 after 1507ms. 

Watchdog: 4 MySQL threads missed a heartbeat

Environment

ProxySQL Version: 3.0.5-centos
Replicas: 3
cpu cores requests: 4 per instance
memory allocation: 3 Gi per instance

(resource usage is very stable and well within bounds)

Deployment Architecture & Traffic Flow

• Traffic Pattern: We operate a single-writer topology. 100% of client traffic is directed to sql01
• Replication: Nodes sql02 and sql03 are maintained strictly for synchronous replication and failover purposes; they do not handle active client queries under normal conditions.
• Impact: Because all traffic is localized to Node 1, any false-positive "OFFLINE" status triggered by the monitor causes an unnecessary and disruptive cluster failover.

Connection Usage Context

Based on our current metrics:

• Client Connections: ~1,300 active connections per ProxySQL instance.
• Backend Connections (In-Use): Peaks of 300 to 700 connections.
• Multiplexing: false (due to use of transaction, medium-long)

Relevant Configuration

We have adjusted several monitor and connection timeouts to mitigate this, but the issue persists on startup:

| Variable                                   | Value     | Description                                        |
|:-------------------------------------------|:----------|:---------------------------------------------------|
| `mysql-threads`                            | **4**     | Number of worker threads.                          |
| `mysql-monitor_threads_min`                | **4**     | Min threads for monitoring tasks.                  |
| `mysql-monitor_threads_max`                | **128**   | Max threads for monitoring tasks.                  |
| `mysql-monitor_galera_healthcheck_timeout` | **1500**  | Max time for a Galera health check (ms).           |
| `mysql-monitor_connect_timeout`            | **1000**  | Timeout for monitor connection establishment (ms). |
| `mysql-monitor_ping_timeout`               | **1500**  | Timeout for monitor ICMP/Ping (ms).                |
| `mysql-monitor_read_only_timeout`          | **1000**  | Timeout for read-only checks (ms).                 |
| `mysql-monitor_query_timeout`              | **1000**  | Timeout for monitor queries (ms).                  |
| `mysql-connect_timeout_server`             | **1000**  | Max TCP connection timeout to backends (ms).       |
| `mysql-connect_timeout_server_max`         | **10000** | Max TCP connection timeout to backends (ms).       |
| `mysql-poll_timeout`                       | **500**   | Timeout for internal poll() calls (ms).            |

Health Log Comparison

1. mysql_server_ping_log (Network Responsiveness)

The network is clear; backends respond in microseconds.

| Hostname      | success_time_us | Latency (ms) | Status    |
|:--------------|:----------------|:-------------|:----------|
| sql02 | 133             | 0.13ms       | Excellent |
| sql01 | 142             | 0.14ms       | Excellent |
| sql01 | 215             | 0.21ms       | Good      |
| sql02 | 233             | 0.23ms       | Good      |
| sql01 | 413             | 0.41ms       | Good      |
| sql01 | 449             | 0.44ms       | Good      |
| sql03 | 1381            | 1.38ms       | Average   |
| sql03 | 1470            | 1.47ms       | Average   |
| sql03 | 1513            | 1.51ms       | Average   |
| sql03 | 1761            | 1.76ms       | Worse     |
| sql03 | 2002            | 2.00ms       | Worse     |
| sql03 | 2672            | 2.67ms       | Worst     |

2. mysql_server_connect_log (TCP Connection)

TCP handshake remains stable (13ms - 35ms).

| Hostname      | connect_success_time_us | Latency (ms) | Status  |
|:--------------|:------------------------|:-------------|:--------|
| sql01 | 13060                   | 13.0ms       | Good    |
| sql02 | 13116                   | 13.1ms       | Good    |
| sql01 | 13256                   | 13.2ms       | Good    |
| sql02 | 13388                   | 13.3ms       | Good    |
| sql01 | 14669                   | 14.6ms       | Good    |
| sql02 | 16837                   | 16.8ms       | Average |
| sql03 | 20358                   | 20.3ms       | Average |
| sql03 | 21564                   | 21.5ms       | Average |
| sql01 | 22278                   | 22.2ms       | Average |
| sql01 | 25691                   | 25.6ms       | Worse   |
| sql03 | 29117                   | 29.1ms       | Worse   |
| sql03 | 35098                   | 35.0ms       | Worst   |

3. mysql_server_galera_log (Monitor Internal Stalling)

Massive discrepancy: Galera success times are up to 100x slower than TCP connects.

| Hostname      | success_time_us | Latency (ms) | Status         |
|:--------------|:----------------|:-------------|:---------------|
| sql02 | 2464            | 2.4ms        | Good           |
| sql02 | 2946            | 2.9ms        | Good           |
| sql01 | 4823            | 4.8ms        | Good           |
| sql01 | 7195            | 7.1ms        | Good           |
| sql03 | 7637            | 7.6ms        | Good           |
| sql01 | 11422           | 11.4ms       | Average        |
| sql03 | 21440           | 21.4ms       | Average        |
| sql01 | 38058           | 38.0ms       | Worse          |
| sql03 | 47764           | 47.7ms       | Worse          |
| sql02 | 102062          | 102.0ms      | Critical Spike |
| sql02 | 106698          | 106.6ms      | Critical Spike |
| sql01 | **(Timeout)**   | **>1500ms**  | **Failed**     |

Analysis & Conclusion

1. Thread startvation: Is there thread starvation? The immediate failure on startup suggests that ProxySQL threads are
   saturated
2. Internal Bottleneck: The discrepancy between pings (0.1ms) and Galera checks (>1500ms) indicates ProxySQL is failing to process results because worker threads are blocked.
3. Potential Contention: With mysql-threads set to 4, the initialization phase likely triggers mutex contention or CPU exhaustion, causing the Watchdog to fire and health checks to time out before the internal scheduler can even register the database's response which is weird cause we do not see any spikes in CPU usage at that time

Request: Please investigate potential lock contention in MySQL_Monitor.cpp during periods of high connection establishment, especially at startup and periods of higher load

[coderabbitai]

🔗 Related PRs

#5416 - Handle late COPY protocol messages after session exits fast_forward mode - P0 [merged]
#5453 - [WIP] multiple changes related to CI [merged]
#5460 - Fix TAP tests for CI migration to per-container hostnames [merged]
#5528 - Run ProxySQL cluster inside single container + CI stability fixes [merged]
#5538 - CI test fixes: cluster sync, plan counts, test group assignments, and diagnostics [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[renecannao]

Hi Thomas,

I believe all analysis and conclusions are incorrect, and I have a completely different theory.

First of all, since mysql_server_ping_log confirms very small latency, the problem is not network and it is not any sort of internal contention in proxysql (otherwise also ping would be slow).
Ping is a simple operation: send a COM_PING and get an OK_Packet , super super easy.
Checking Galera status: send a COM_QUERY to check internal status of Galera and wait for a reply. If this takes long time, I think it is obvious where the time is spent: inside Galera. You can easily reproduce this yourself outside of ProxySQL: check how fast is Galera in returning its status.
And it is no surprise that this becomes more evident during higher load: the server is busier and it takes more time to generate status results.

Backend Connections (In-Use): Peaks of 300 to 700 connections.

This is high! It is easy to speculate that there is contention on the backend.

Also mysql_server_ping_log show unstable backed response time, and unless you have unstable network this is another warning that something is wrong with the backend.

Now, the watchdog error , this is the not trivial issue.
How often does this happen? Does it recover itself?
ProxySQL performs async network operations, with only one exception: DNS lookup.
If DNS lookup takes time this would explain two issues at once:

• watchdog error , because DNS lookups are blocking
• more instability during start time , as proxysql didn't cache DNS entries yet.

Question: if a monitoring tool reports that a monitored target is unresponsive, why do you investigate the monitoring tool and not the monitored target?
ProxySQL is ringing the bell: "something is wrong with the backend" . Please do not ignore the alert.
Please don't shoot the messenger.

[ThomasVerhoeven1998]

The monitored target has been evaluate multiple times by a DBA expert. We are running a 3 node Percona XtraDB cluster with each 16 cpu cores and 128 GB of memory. Cluster resources usage is very stable.

We have looked at other metrics, mainly WSREP_FLOW_CONTROL_PAUSED and WSREP_LOCAL_RECV_QUEUE_AVG. There is sometimes flow control happing on the replication nodes when there is heavy write

Query execution times are as following (in ms):

• P50: 3 ms
• P90: 15 ms
• P99: 38 ms (high load)

It mostly during startup that these watchdog error and timeouts are firing. The watchdogs do recover indeed!

Looking at the source code, we notice the following:

In MySQL_Monitor::run(), the initial monitor worker thread pool is created with num_threads = 2 (hardcoded in the constructor at line 1108). The scaling to mysql_thread___monitor_threads_min only fires inside the main loop's config-version-change handler — which never triggers if the configuration was already applied at startup (the version seen in the constructor equals the version seen on the first loop iteration).

What we believe is happening at startup?

1. Startup, 2 threads: All monitor types (connect, ping, read-only, galera) share 2 worker threads. All tasks need new connections (none cached), and create_new_connection() is synchronous (13–35ms per connection). With 12+ tasks queued and 2 threads, the last task can wait 200ms+ for a thread.

2. Galera timeouts fire: Even though t1 is reset after connection establishment, scheduling jitter in the poll() loop (10ms granularity) accumulates when the OS preempts threads under load. Galera checks start crossing the 1500ms mark.

3. update_galera_set_offline() holds GloAdmin->mysql_servers_wrlock() for extended time (multiple SQLite queries + commit()) — this blocks all 4 MySQL worker threads waiting for the lock, triggering the Watchdog.

4. Cascade: More timeouts occur during the Watchdog stall → more offline calls → more lock contention.

We're not trying to "shoot" anyone, just trying to understand what's happening.

I'll conduct further tests massively reducing the number of connections by scaling our microservices lower and see how it behaves.

When running the exact query the monitor runs:

SET @t1 = NOW(6);

SELECT
    (SELECT VARIABLE_VALUE FROM performance_schema.global_status  WHERE VARIABLE_NAME='WSREP_LOCAL_STATE')         AS wsrep_local_state,
    @@read_only                                                                                                      AS read_only,
    (SELECT VARIABLE_VALUE FROM performance_schema.global_status  WHERE VARIABLE_NAME='WSREP_LOCAL_RECV_QUEUE')    AS wsrep_local_recv_queue,
    @@wsrep_desync                                                                                                   AS wsrep_desync,
    @@wsrep_reject_queries                                                                                           AS wsrep_reject_queries,
    @@wsrep_sst_donor_rejects_queries                                                                               AS wsrep_sst_donor_rejects_queries,
    (SELECT VARIABLE_VALUE FROM performance_schema.global_status  WHERE VARIABLE_NAME='WSREP_CLUSTER_STATUS')      AS wsrep_cluster_status,
    (SELECT COALESCE(MAX(VARIABLE_VALUE),'DISABLED') FROM performance_schema.global_variables WHERE variable_name='pxc_maint_mode') AS pxc_maint_mode;

SELECT TIMESTAMPDIFF(MICROSECOND, @t1, NOW(6)) / 1000 AS health_check_ms;

I do see indeed the health_check_ms is around 1200 ms. Very surprised by this to be honest. Thanks for the call on that one!

[renecannao]

These graphs are meaningless in the context of this investigation. This what is relevant:

You can easily reproduce this yourself outside of ProxySQL: check how fast is Galera in returning its status.

You need to measure what ProxySQL is measing.

update_galera_set_offline() holds GloAdmin->mysql_servers_wrlock() for extended time (multiple SQLite queries + commit()) — this blocks all 4 MySQL worker threads waiting for the lock, triggering the Watchdog.

GloAdmin->mysql_servers_wrlock() is not in the critical path of MySQL worker.
And even if it was, this should take a fraction of millisecond, not trigger a watchdog error.

We're not trying to "shoot" anyone, just trying to understand what's happening.

Allow me to disagree :-)
So far you are digging into ProxySQL code, instead of verifying if its claims are correct.
Did you check how long it takes to run Galera checks? It doesn't seem.

[renecannao]

I do see indeed the health_check_ms is around 1200 ms. Very surprised by this to be honest. Thanks for the call on that one!

You are welcome.
❤️