Unit Workload Agent Address Ports Message
mysql-k8s/0* active idle 10.1.242.207 Primary
mysql-k8s/1 active idle 10.1.242.215
mysql-k8s/2 maintenance executing 10.1.242.218 Initialising mysqld
unit-mysql-k8s-0: 14:12:27 INFO unit.mysql-k8s/0.juju-log __main__:Configuring initialized mysqld
unit-mysql-k8s-0: 14:12:27 DEBUG unit.mysql-k8s/0.juju-log charms.mysql.v0.mysql:Missing MySQL role charmed_router
...
unit-mysql-k8s-1: 14:12:39 INFO unit.mysql-k8s/1.juju-log __main__:Configuring initialized mysqld
unit-mysql-k8s-1: 14:12:39 DEBUG unit.mysql-k8s/1.juju-log charms.mysql.v0.mysql:Missing MySQL role charmed_router
unit-mysql-k8s-1: 14:12:39 DEBUG unit.mysql-k8s/1.juju-log charms.mysql.v0.mysql:Configuring Router role for mysql-k8s-1.mysql-k8s-endpoints.testing.svc.cluster.local.
unit-mysql-k8s-0: 14:12:39 DEBUG unit.mysql-k8s/0.juju-log root:Re-scanning InnoDB cluster test_cluster
unit-mysql-k8s-2: 14:12:40 DEBUG unit.mysql-k8s/2.juju-log mysql_k8s_helpers:MySQL connection possible
unit-mysql-k8s-2: 14:12:40 INFO unit.mysql-k8s/2.juju-log __main__:Configuring initialized mysqld
unit-mysql-k8s-2: 14:12:40 ERROR unit.mysql-k8s/2.juju-log root:Failed to search instance roles with name_pattern='%router'
unit-mysql-k8s-2: 14:12:40 DEBUG unit.mysql-k8s/2.juju-log mysql_k8s_helpers:Resetting MySQL data directory.
unit-mysql-k8s-2: 14:12:40 ERROR unit.mysql-k8s/2.juju-log root:Uncaught exception while in charm code:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/src/mysql_k8s_executor.py", line 187, in execute_sql
stdout, _ = process.wait_output()
^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/venv/lib/python3.12/site-packages/ops/pebble.py", line 1857, in wait_output
raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
ops.pebble.ExecError: non-zero exit code 1 executing ['/usr/bin/mysqlsh', '--json=raw', '--save-passwords=never', '--passwords-from-stdin', '--socket=/var/run/mysqld/mysqld.sock', '--user=charmed-operator', '--sql', '--execute', "SELECT user, host FROM mysql.user WHERE user LIKE '%router' AND authentication_string=''"], stdout='{"prompt":"Please provide the password for \'charmed-operator@/var%2Frun%2Fmysqld%2Fmysqld.sock\': ","type":"password","defaultValue":""}\n', stderr="Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory\nMySQL Error 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)\n"
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/lib/charms/mysql/v0/mysql.py", line 1240, in configure_mysql_router_roles
router_roles = self._instance_client_sock.search_instance_roles("%router")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/venv/lib/python3.12/site-packages/mysql_shell/clients/instance.py", line 525, in search_instance_roles
rows = self._executor.execute_sql(query)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/src/mysql_k8s_executor.py", line 191, in execute_sql
raise ExecutionError(err) from exc
mysql_shell.executors.errors.runtime.ExecutionError: None
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/src/charm.py", line 1110, in <module>
main(MySQLOperatorCharm)
File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/venv/lib/python3.12/site-packages/ops/main.py", line 39, in main
return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 507, in main
manager.run()
File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 491, in run
self._emit()
File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 426, in _emit
self._emit_charm_event(self.dispatcher.event_name)
File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 470, in _emit_charm_event
event_to_emit.emit(*args, **kwargs)
File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 337, in emit
framework._emit(event)
File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 910, in _emit
self._reemit(event_path)
File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 1028, in _reemit
custom_handler(event)
File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/src/charm.py", line 837, in _on_mysql_pebble_ready
self._configure_instance(container)
File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/src/charm.py", line 732, in _configure_instance
self._mysql.configure_mysql_router_roles()
File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/lib/charms/mysql/v0/mysql.py", line 1243, in configure_mysql_router_roles
raise MySQLConfigureMySQLRolesError() from e
charms.mysql.v0.mysql.MySQLConfigureMySQLRolesError
Notice how units 0 and 1 flow normally, while unit 2 fails.
Part of the flakiness of our tests comes from situations that look like these
Example logs:
Notice how units 0 and 1 flow normally, while unit 2 fails.
The offending code is here:
mysql-operators/kubernetes/src/charm.py
Lines 730 to 735 in 496a6ae
And here are some examples from the same GHA run: