Skip to content

Run ProxySQL cluster inside single container + CI stability fixes #5528

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
renecannao merged 15 commits into from
Mar 23, 2026
Merged

Run ProxySQL cluster inside single container + CI stability fixes #5528

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
6fcb8fd
Run ProxySQL cluster nodes inside a single container
renecannao Mar 23, 2026
8c67073
Redirect coverage generation output to dedicated log file
renecannao Mar 23, 2026
0590803
Limit code coverage to proxysql source directories only
renecannao Mar 23, 2026
ce92a2a
Redirect coverage generation output to log file in run-multi-group.bash
renecannao Mar 23, 2026
359e6be
Update debug filters in infra-mysql57 infra-config.sql
renecannao Mar 23, 2026
6769ea4
Make legacy/pre-proxysql.bash a no-op
renecannao Mar 23, 2026
801a443
Split clickhouse tests into separate legacy-clickhouse group
renecannao Mar 23, 2026
5a868d3
Default AUTO_CLEANUP=1 in run-multi-group.bash
renecannao Mar 23, 2026
8705685
Address PR review comments: port collisions, gcov, loop bounds, fastcov
renecannao Mar 23, 2026
35a9bfb
Use 'proxysql' hostname in proxysql_servers instead of 127.0.0.1
renecannao Mar 23, 2026
cc9a041
Add connection details to noise thread failure logs
renecannao Mar 23, 2026
e299de5
Add step-by-step diagnostics to reg_test__ssl_client_busy_wait-t
renecannao Mar 23, 2026
4837642
Fix port conflicts: assign unique pgsql/sqlite ports to cluster nodes
renecannao Mar 23, 2026
98b56ec
Move cluster node pgsql/sqlite ports to 7xxx range to avoid restapi c…
renecannao Mar 23, 2026
c84f411
Fix pgsql admin port: set pgsql_ifaces in admin_variables, not pgsql_…
renecannao Mar 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 19 additions & 13 deletions test/infra/control/check_all_nodes.bash
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,22 +1,28 @@
#!/usr/bin/env bash
set -e
set -o pipefail
#
# Scheduler script: polls all ProxySQL cluster nodes to keep them active.
# Runs inside the ProxySQL container where all nodes are on 127.0.0.1.
#
# Port scheme:
# Primary: 6032
# proxy-node1: 6042
# proxy-node2: 6052
# ...
# proxy-node9: 6122

TABLES=(mysql_servers mysql_users mysql_query_rules mysql_query_rules_fast_routing global_variables proxysql_servers mysql_galera_hostgroups mysql_group_replication_hostgroups mysql_replication_hostgroups mysql_hostgroup_attributes)

ALL_TABLES=()

for i in ; do
ALL_TABLES+=()
ALL_TABLES+=("runtime_")
for i in ${!TABLES[@]}; do
ALL_TABLES+=(${TABLES[$i]})
ALL_TABLES+=("runtime_"${TABLES[$i]})
done
Comment on lines +16 to 19
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist bot Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This loop can be written more idiomatically using for item in "${array[@]}" syntax, which is generally more readable and less error-prone than iterating over indices.

for table in "${TABLES[@]}"; do
    ALL_TABLES+=("${table}")
    ALL_TABLES+=("runtime_${table}")
done


# The nodes in our containerized cluster
NODES="proxysql proxy-node1 proxy-node2 proxy-node3 proxy-node4 proxy-node5 proxy-node6 proxy-node7 proxy-node8 proxy-node9"
# Primary + up to 9 nodes, all on 127.0.0.1 with different ports
PORTS="6032 6042 6052 6062 6072 6082 6092 6102 6112 6122"

for host in ; do
# Use radmin/radmin for remote access between containers
for i in ; do
echo "SELECT COUNT(*) FROM ;"
done | mysql -u radmin -pradmin -h -P 6032 > /dev/null &
for port in ${PORTS}; do
for i in ${!ALL_TABLES[@]}; do
echo "SELECT COUNT(*) FROM ${ALL_TABLES[$i]};"
done | mysql -u admin -padmin -h 127.0.0.1 -P ${port} > /dev/null 2>&1 &
done
Comment on lines +24 to 28
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist bot Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The inner loop generating SQL queries can be made more efficient and readable by using printf to generate all queries at once, avoiding the shell loop. Also, it's good practice to quote shell variables like ${port} when they are used as arguments.

for port in ${PORTS}; do
    printf 'SELECT COUNT(*) FROM %s;\n' "${ALL_TABLES[@]}" | mysql -u admin -padmin -h 127.0.0.1 -P "${port}" > /dev/null 2>&1 &
done

13 changes: 10 additions & 3 deletions test/infra/control/env-isolated.bash
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,10 +59,17 @@ export TAP_DEPS="${WORKSPACE}/test/tap/tap"
export TEST_DEPS_PATH="${WORKSPACE}/test-scripts/deps"
export TEST_DEPS="${TEST_DEPS_PATH}"

# Cluster Nodes
# Cluster Nodes — all run inside the ProxySQL container on different ports
# Port scheme: proxy-node1=6042, proxy-node2=6052, ..., proxy-node9=6122
# From the test-runner container, reach them via the proxysql hostname
NUM_CLUSTER_NODES=${PROXYSQL_CLUSTER_NODES:-9}
if [[ "${SKIP_CLUSTER_START}" == "1" ]] || [[ "${SKIP_CLUSTER_START}" == "true" ]]; then
NUM_CLUSTER_NODES=0
fi
CLUSTER_NODES=""
for i in $(seq 1 9); do
CLUSTER_NODES="${CLUSTER_NODES}proxy-node${i}:6032,"
for i in $(seq 1 ${NUM_CLUSTER_NODES}); do
PORT=$((6032 + i * 10))
CLUSTER_NODES="${CLUSTER_NODES}proxysql:${PORT},"
done
Comment on lines +62 to 73
Copy link

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TAP_CLUSTER_NODES is still built with seq 1 9 regardless of PROXYSQL_CLUSTER_NODES / SKIP_CLUSTER_START. Since the cluster size is now configurable, this can make tests attempt to connect to nodes that weren’t started (or miss nodes when >9). Please derive the loop bound from PROXYSQL_CLUSTER_NODES (and produce an empty list when cluster start is skipped).

Copilot uses AI. Check for mistakes.
export TAP_CLUSTER_NODES=${CLUSTER_NODES%,}

Expand Down
22 changes: 13 additions & 9 deletions test/infra/control/run-multi-group.bash
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ set -euo pipefail
# PARALLEL_JOBS=4 # Max parallel groups (default: unlimited)
# TIMEOUT_MINUTES=60 # Hard timeout per group (default: 60)
# EXIT_ON_FIRST_FAIL=0 # Stop on first failure (default: 0)
# AUTO_CLEANUP=0 # Auto cleanup successful groups (default: 0)
# AUTO_CLEANUP=1 # Auto cleanup successful groups (default: 1)
# SKIP_CLUSTER_START=1 # Skip ProxySQL cluster initialization (default: 0)
# COVERAGE=1 # Enable code coverage collection (default: 0)
# TAP_USE_NOISE=1 # Enable noise injection for race condition testing (default: 0)
Expand All @@ -35,7 +35,7 @@ TAP_GROUPS="${TAP_GROUPS:-}"
PARALLEL_JOBS="${PARALLEL_JOBS:-2}" # Default: 2 parallel groups
TIMEOUT_MINUTES="${TIMEOUT_MINUTES:-60}"
EXIT_ON_FIRST_FAIL="${EXIT_ON_FIRST_FAIL:-0}"
AUTO_CLEANUP="${AUTO_CLEANUP:-0}"
AUTO_CLEANUP="${AUTO_CLEANUP:-1}"
SKIP_CLUSTER_START="${SKIP_CLUSTER_START:-0}"
COVERAGE="${COVERAGE:-0}"
TAP_USE_NOISE="${TAP_USE_NOISE:-0}"
Expand Down Expand Up @@ -354,27 +354,30 @@ if [ "${COVERAGE}" -eq 1 ]; then

if [ -n "${COVERAGE_FILES}" ]; then
COMBINED_INFO="${COMBINED_COVERAGE_DIR}/combined-coverage.info"
COVERAGE_LOG="${COMBINED_COVERAGE_DIR}/coverage-generation.log"
echo ">>> Combining coverage reports into: ${COMBINED_INFO}"
echo ">>> Coverage generation log: ${COVERAGE_LOG}"

# Run coverage combination in container (tools may not be on host)
docker run --rm \
-v "${WORKSPACE}:${WORKSPACE}" \
-e COVERAGE_FILES="${COVERAGE_FILES}" \
-e COMBINED_INFO="${COMBINED_INFO}" \
-e COVERAGE_LOG="${COVERAGE_LOG}" \
proxysql-ci-base:latest \
bash -c '
set -e
if command -v fastcov >/dev/null 2>&1; then
fastcov -b -l -C ${COVERAGE_FILES} -o "${COMBINED_INFO}" 2>&1 || {
echo ">>> WARNING: fastcov combine failed, trying lcov..."
fastcov -b -l -C ${COVERAGE_FILES} -o "${COMBINED_INFO}" >> "${COVERAGE_LOG}" 2>&1 || {
echo ">>> WARNING: fastcov combine failed, trying lcov..." >> "${COVERAGE_LOG}"
if command -v lcov >/dev/null 2>&1; then
FIRST_FILE=true
for info_file in ${COVERAGE_FILES}; do
if [ "${FIRST_FILE}" = true ]; then
cp "${info_file}" "${COMBINED_INFO}"
FIRST_FILE=false
else
lcov -a "${COMBINED_INFO}" -a "${info_file}" -o "${COMBINED_INFO}".tmp 2>/dev/null && \
lcov -a "${COMBINED_INFO}" -a "${info_file}" -o "${COMBINED_INFO}".tmp >> "${COVERAGE_LOG}" 2>&1 && \
mv "${COMBINED_INFO}".tmp "${COMBINED_INFO}"
fi
done
Expand All @@ -387,15 +390,15 @@ if [ "${COVERAGE}" -eq 1 ]; then
cp "${info_file}" "${COMBINED_INFO}"
FIRST_FILE=false
else
lcov -a "${COMBINED_INFO}" -a "${info_file}" -o "${COMBINED_INFO}".tmp 2>/dev/null && \
lcov -a "${COMBINED_INFO}" -a "${info_file}" -o "${COMBINED_INFO}".tmp >> "${COVERAGE_LOG}" 2>&1 && \
mv "${COMBINED_INFO}".tmp "${COMBINED_INFO}"
fi
done
else
echo ">>> ERROR: Neither fastcov nor lcov available"
exit 1
fi
' || echo ">>> WARNING: Coverage combination failed"
' || echo ">>> WARNING: Coverage combination failed (see ${COVERAGE_LOG})"

if [ -f "${COMBINED_INFO}" ]; then
echo ">>> Combined coverage report: ${COMBINED_INFO}"
Expand All @@ -409,11 +412,12 @@ if [ "${COVERAGE}" -eq 1 ]; then
-v "${WORKSPACE}:${WORKSPACE}" \
-e COMBINED_INFO="${COMBINED_INFO}" \
-e COMBINED_HTML="${COMBINED_HTML}" \
-e COVERAGE_LOG="${COVERAGE_LOG}" \
proxysql-ci-base:latest \
bash -c '
if command -v genhtml >/dev/null 2>&1; then
genhtml --branch-coverage --ignore-errors negative,source --synthesize-missing "${COMBINED_INFO}" --output-directory "${COMBINED_HTML}" 2>&1 || \
echo ">>> WARNING: HTML generation failed"
genhtml --branch-coverage --ignore-errors negative,source --synthesize-missing "${COMBINED_INFO}" --output-directory "${COMBINED_HTML}" >> "${COVERAGE_LOG}" 2>&1 || \
echo ">>> WARNING: HTML generation failed (see ${COVERAGE_LOG})"
else
echo ">>> WARNING: genhtml not available"
fi
Expand Down
13 changes: 9 additions & 4 deletions test/infra/control/run-tests-isolated.bash
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -264,13 +264,15 @@ docker run \
if command -v fastcov >/dev/null 2>&1; then
mkdir -p \"\${COVERAGE_REPORT_DIR}\"
local coverage_file=\"\${COVERAGE_REPORT_DIR}/\${INFRA_ID}.info\"
local coverage_log=\"\${COVERAGE_REPORT_DIR}/coverage-generation.log\"
echo \">>> Generating coverage report: \${coverage_file}\"
echo \">>> Coverage generation log: \${coverage_log}\"
local nproc_val=\$(nproc)

# Copy .gcno files to /gcov so fastcov can find both .gcno and .gcda together
# This avoids race conditions when multiple groups run in parallel
if [ -d \"/gcov\" ] && [ \"\$(ls -A /gcov 2>/dev/null)\" ]; then
echo \">>> Preparing coverage data directory...\"
echo \">>> Preparing coverage data directory...\" >> \"\${coverage_log}\" 2>&1
cd \"\${WORKSPACE}\" && find . -path './ci_infra_logs' -prune -o -name '*.gcno' -type f -print | while read gcno; do
target=\"/gcov/\${gcno#./}\"
target_dir=\"\$(dirname \"\$target\")\"
Expand All @@ -279,7 +281,9 @@ docker run \
done
echo \">>> Running fastcov on /gcov...\"
cd /gcov
fastcov -b -j\"\${nproc_val}\" --process-gcno -l -e /usr/include/ -e \"\${WORKSPACE}/test/tap/tests\" -e \"\${WORKSPACE}/deps/\" -d . -o \"\${coverage_file}\" 2>&1 || echo \">>> WARNING: Coverage generation failed\"
fastcov -b -j\"\${nproc_val}\" --process-gcno -l \
--include \"\${WORKSPACE}/include/\" \"\${WORKSPACE}/lib/\" \"\${WORKSPACE}/src/\" \"\${WORKSPACE}/test/\" \
-d . -o \"\${coverage_file}\" >> \"\${coverage_log}\" 2>&1 || echo \">>> WARNING: Coverage generation failed (see \${coverage_log})\"
Comment thread
coderabbitai[bot] marked this conversation as resolved.
else
echo \">>> WARNING: /gcov directory is empty or missing, skipping coverage\"
fi
Expand All @@ -290,11 +294,12 @@ docker run \
if command -v genhtml >/dev/null 2>&1; then
local html_dir=\"\${COVERAGE_REPORT_DIR}/html\"
mkdir -p \"\${html_dir}\"
genhtml --branch-coverage --ignore-errors negative,source --synthesize-missing \"\${coverage_file}\" --output-directory \"\${html_dir}\" 2>&1 || echo \">>> WARNING: HTML generation failed\"
echo \">>> Generating HTML coverage report...\"
genhtml --branch-coverage --ignore-errors negative,source --synthesize-missing \"\${coverage_file}\" --output-directory \"\${html_dir}\" >> \"\${coverage_log}\" 2>&1 || echo \">>> WARNING: HTML generation failed (see \${coverage_log})\"
[ -f \"\${html_dir}/index.html\" ] && echo \">>> HTML coverage report: \${html_dir}/index.html\"
fi
else
echo \">>> WARNING: Coverage info file not generated\"
echo \">>> WARNING: Coverage info file not generated (see \${coverage_log})\"
fi
else
echo \">>> WARNING: fastcov not found in container, skipping coverage collection\"
Expand Down
Loading
Loading