test_read_only_actions_offline_hard_servers-t leaves hostgroup 1300 empty, breaking subsequent tests

[renecannao]

Problem

test_read_only_actions_offline_hard_servers-t empties hostgroup 1300 during its setup, fails to execute its test scenario, and doesn't restore the server configuration on cleanup. All subsequent tests that rely on hostgroup 1300 fail with:

Max connect timeout reached while reaching hostgroup 1300 after 10000ms

Affected tests (cascading failures in legacy-g4)

First pass:

• test_throttle_max_bytes_per_second_to_client-t
• test_unsupported_queries-t
• test_warnings-t

Second pass (all tests_with_deps/deprecate_eof_support):

• deprecate_eof_cache-t
• eof_cache_mixed_flags-t
• eof_conn_options_check-t
• eof_fast_forward-t
• eof_mixed_flags_queries-t
• eof_packet_mixed_queries-t
• ok_packet_mixed_queries-t

Evidence from debug dump

dump_hostgroup_debug() output at test_warnings-t start:

=== Hostgroup 1300 Debug Dump ===
runtime_mysql_servers (hg=1300):
  (no servers in hostgroup 1300)
stats_mysql_connection_pool (hg=1300):
  (no pool entries for hostgroup 1300)
=== End Hostgroup 1300 Debug Dump ===

While hostgroup 0 (fallback) still has servers:

hg=0 host=mysql1.infra-mysql57 port=3306 status=ONLINE

Root cause

test_read_only_actions_offline_hard_servers-t replaces mysql_servers with its own test configuration (only hostgroups 0/1), then fails with "hostgroup 1300 timeout" because its own test scenario can't work with the new infra's hostgroup layout. The cleanup path (goto cleanup) attempts LOAD MYSQL SERVERS FROM DISK but doesn't verify the restore succeeded.

Fix needed

The test's cleanup must reliably restore hostgroup 1300 servers, regardless of whether the test passes or fails. Options:

1. Save and restore mysql_servers content explicitly (not via DISK)
2. Verify after LOAD MYSQL SERVERS FROM DISK that hostgroup 1300 has servers
3. Adapt the test to work with hostgroup 1300 instead of only 0/1

[coderabbitai]

🔗 Related PRs

#5445 - Rollback test_binlog_reader-t.cpp to tag v3.0.5 [merged]
#5460 - Fix TAP tests for CI migration to per-container hostnames [merged]
#5506 - HostGroups Manager unit tests (Phase 2.6) [merged]
#5520 - Add SKIP_PROXYSQL for infrastructure-free test groups (unit tests) [merged]
#5528 - Run ProxySQL cluster inside single container + CI stability fixes [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[renecannao]

Root cause confirmed: cluster sync saves corrupted config to disk

The chain of events:

1. test_read_only_actions_offline_hard_servers-t replaces mysql_servers with only hostgroup 0/1 entries and LOADs to runtime
2. Cluster sync detects the checksum change on the primary
3. A cluster node pulls the new config (no hostgroup 1300) and saves it to disk — because cluster_mysql_servers_save_to_disk=true (default)
4. The test fails and its cleanup does LOAD MYSQL SERVERS FROM DISK — but disk is now corrupted
5. proxysql-tester.py's reconfigure_proxysql() also loads from disk before each subsequent test — still corrupted
6. All subsequent tests needing hostgroup 1300 fail

Evidence: reconfigure_proxysql() ran before test_throttle (logged at Timed 'reconfigure_proxysql' : 0.002s) but the debug dump shows hostgroup 1300 is empty. The disk was corrupted by cluster sync between test_read_only's LOAD TO RUNTIME and its cleanup.

Fix: The test must set cluster_mysql_servers_save_to_disk=false before modifying mysql_servers, and restore it at cleanup. This prevents the cluster from persisting the test's temporary configuration to disk.

The test_cluster1-t already does this correctly:

for module in modules:
    q = "SET admin-cluster_" + module + "_save_to_disk='false'"

[renecannao]

Circular cluster sync confirmed with log evidence

The ProxySQL error log shows the exact chain:

Timeline:

• 17:52:08 — test_read_only_actions_offline_hard_servers-t modifies mysql_servers (removes hostgroup 1300), LOADs to runtime. The primary's epoch advances.
• 17:52:08-17:52:56 — Cluster nodes sync the corrupted config FROM the primary (normal sync behavior). Node 6042 now has the corrupted config with a newer epoch.
• 17:52:56 — Test fails, cleanup does LOAD MYSQL SERVERS FROM DISK on primary — restores original config. But the epoch of this restored config is older than what the nodes have.
• 17:55:08-17:55:49 — reconfigure_proxysql() runs before subsequent tests, does LOAD FROM DISK + LOAD TO RUNTIME. Primary has the correct config again. But nodes still have the corrupted config with a newer epoch.
• 17:55:55 — Primary detects node 6042 has a DIFFERENT checksum (0x7C0A67D5FF50CD51) with a newer epoch (1774288555 > 1774288549). After diff_check=3, primary decides to sync FROM node:

Cluster: detected a peer proxysql:6042 with mysql_servers version 28, epoch 1774288555, diff_check 3.
Own version: 276, epoch: 1774288549. Proceeding with remote sync

• 17:55:57 — Primary fetches corrupted MySQL Servers from node 6042 and saves to disk:

Cluster: Fetching 'MySQL Servers' from peer proxysql:6042 started. Expected checksum 0x7C0A67D5FF50CD51
Cluster: Saving to disk MySQL Servers v2 from peer proxysql:6042

Root cause: LOAD FROM DISK restores the correct server config but doesn't advance the epoch beyond what the cluster nodes have. The cluster considers the node's corrupted config to be "newer" because it has a higher epoch, and syncs it back to the primary, overwriting the restored config and saving it to disk.

This is a design issue: there's no way for a LOAD FROM DISK operation to take precedence over cluster sync when the nodes have a newer epoch with corrupted data.

[renecannao]

Deeper root cause: monitor on cluster nodes creates divergent checksums

Evidence from node 6042's log (proxy-node1):

17:55:52 — Node 6042 syncs mysql_servers_v2 from primary (proxysql:6032), gets checksum 0x30972347F0ACCCBE with epoch 1774288549. It loads to runtime. Then the monitor on node 6042 processes mysql_replication_hostgroups, checks read_only on backends, and moves servers between writer/reader hostgroups:

[WARNING] Removed server at address ..., hostgroup 1701, address mariadb3.infra-mariadb10 port 3306. Setting status OFFLINE HARD
[WARNING] Removed server at address ..., hostgroup 1701, address mariadb2.infra-mariadb10 port 3306. Setting status OFFLINE HARD
[WARNING] Removed server at address ..., hostgroup 1700, address mariadb1.infra-mariadb10 port 3306. Setting status OFFLINE HARD

This changes node 6042's runtime mysql_servers, producing a NEW checksum (0x7C0A67D5FF50CD51) with a new epoch (1774288555).

17:55:55 — Primary sees node 6042 with the new (divergent) checksum and newer epoch. After diff_check=3, it syncs FROM node 6042 — overwriting its own correct config with the monitor-modified version and saving to disk.

The issue: The monitor on cluster nodes independently modifies mysql_servers (moving servers between writer/reader hostgroups based on read_only checks). This creates checksums that diverge from the primary. Since these changes have a newer epoch, the primary considers them authoritative and syncs from the node.

This is not a test cleanup issue — it's a fundamental interaction between the cluster monitor and cluster sync. The monitor on satellite/core nodes should not modify mysql_servers in a way that causes the primary to sync from them, or the primary should not sync mysql_servers from nodes whose changes came from the monitor rather than from admin commands.

[renecannao]

Root cause confirmed: cluster_mysql_servers_sync_algorithm=1 (default) syncs monitor changes

The default value of admin-cluster_mysql_servers_sync_algorithm is 1 (runtime_mysql_servers_and_mysql_servers_v2), which syncs BOTH configured servers AND runtime (monitor-modified) servers.

The enum values are:

• 1 = runtime_mysql_servers_and_mysql_servers_v2 — syncs runtime changes including monitor moves (the default, and the cause of this bug)
• 2 = mysql_servers_v2 — syncs only user-configured changes (safe from monitor)
• 3 = auto_select — picks 2 when monitor is enabled, 1 when monitor is disabled

When the monitor on node 6042 moves MariaDB servers between writer/reader hostgroups (due to read_only checks), this creates a runtime checksum change with a newer epoch. With algorithm 1, the primary considers this a valid change and syncs FROM the node, overwriting its own config and saving to disk.

Fix for CI: Set cluster_mysql_servers_sync_algorithm=3 (auto_select) or 2 (v2-only) during cluster initialization. Since the monitor is enabled, auto_select will correctly use the v2 algorithm that ignores monitor-triggered runtime changes.

Note: The default of 1 may be problematic in production clusters too — any cluster node's monitor independently modifying servers could trigger circular sync. Consider changing the default to 3 (auto_select).