Integration with ntp charm broken

[gustavosr98]

Steps to reproduce

Deploy mongodb charm
Relate it to ntp charm

Expected behavior

Not to error

Actual behavior

Mongodb charm seems to be breaking /etc/environment file

ubuntu@juju-eba345-mongodb-0:~$ vi /etc/environment 
ubuntu@juju-eba345-mongodb-0:~$ source /etc/environment 
--replSet=mongodb: command not found

MONGOD_ARGS variable seems to be missing quotes in the beginning and end
Manually editing the file to add quotes workarounds the issue

# juju ssh mongodb/0 -- cat /etc/environment

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
MONGOD_ARGS=--bind_ip_all --replSet=mongodb --dbpath=/var/snap/charmed-mongodb/common/var/lib/mongodb --port=27017 --setParameter processUmask=037 --logRotate reopen --logappend --logpath=/var/snap/charmed-mongodb/common/var/log/m
ongodb/mongodb.log --auditDestination=file --auditFormat=JSON --auditPath=/var/snap/charmed-mongodb/common/var/log/mongodb/audit.log --auth --tlsCAFile=/var/snap/charmed-mongodb/current/etc/mongod/external-ca.crt --tlsCertificateKe
yFile=/var/snap/charmed-mongodb/current/etc/mongod/external-cert.pem --tlsMode=preferTLS --tlsDisabledProtocols=TLS1_0,TLS1_1 --clusterAuthMode=x509 --tlsAllowInvalidCertificates --tlsClusterCAFile=/var/snap/charmed-mongodb/current
/etc/mongod/internal-ca.crt --tlsClusterFile=/var/snap/charmed-mongodb/current/etc/mongod/internal-cert.pem

NTP charm now breaks

unit-ntp-0: 18:28:11 ERROR unit.ntp/0.juju-log Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ntp-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-ntp-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-ntp-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-ntp-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-ntp-0/charm/reactive/ntp.py", line 116, in install
    fetch.apt_install(pkgs, fatal=True)
  File "/var/lib/juju/agents/unit-ntp-0/.venv/lib/python3.10/site-packages/charmhelpers/fetch/ubuntu.py", line 369, in apt_install
    _run_apt_command(cmd, fatal, quiet=quiet)
  File "/var/lib/juju/agents/unit-ntp-0/.venv/lib/python3.10/site-packages/charmhelpers/fetch/ubuntu.py", line 945, in _run_apt_command
    _run_with_retries(
  File "/var/lib/juju/agents/unit-ntp-0/.venv/lib/python3.10/site-packages/charmhelpers/fetch/ubuntu.py", line 903, in _run_with_retries
    env = get_apt_dpkg_env()
  File "/var/lib/juju/agents/unit-ntp-0/.venv/lib/python3.10/site-packages/charmhelpers/fetch/ubuntu.py", line 1003, in get_apt_dpkg_env
    'PATH': get_system_env('PATH', '/usr/sbin:/usr/bin:/sbin:/bin')}
  File "/var/lib/juju/agents/unit-ntp-0/.venv/lib/python3.10/site-packages/charmhelpers/core/host.py", line 1295, in get_system_env
    output = subprocess.check_output(
  File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['env', '-i', '/bin/bash', '-c', 'set -a && source /etc/environment && env']' returned non-zero exit status 127.

unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install Traceback (most recent call last):
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install   File "/var/lib/juju/agents/unit-ntp-0/charm/hooks/install", line 22, in <module>
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install     main()
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install   File "/var/lib/juju/agents/unit-ntp-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install     bus.dispatch(restricted=restricted_mode)
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install   File "/var/lib/juju/agents/unit-ntp-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install     _invoke(other_handlers)
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install   File "/var/lib/juju/agents/unit-ntp-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install     handler.invoke()
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install   File "/var/lib/juju/agents/unit-ntp-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install     self._action(*args)
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install   File "/var/lib/juju/agents/unit-ntp-0/charm/reactive/ntp.py", line 116, in install
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install     fetch.apt_install(pkgs, fatal=True)
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install   File "/var/lib/juju/agents/unit-ntp-0/.venv/lib/python3.10/site-packages/charmhelpers/fetch/ubuntu.py", line 369, in apt_install
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install     _run_apt_command(cmd, fatal, quiet=quiet)
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install   File "/var/lib/juju/agents/unit-ntp-0/.venv/lib/python3.10/site-packages/charmhelpers/fetch/ubuntu.py", line 945, in _run_apt_command
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install     _run_with_retries(
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install   File "/var/lib/juju/agents/unit-ntp-0/.venv/lib/python3.10/site-packages/charmhelpers/fetch/ubuntu.py", line 903, in _run_with_retries
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install     env = get_apt_dpkg_env()
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install   File "/var/lib/juju/agents/unit-ntp-0/.venv/lib/python3.10/site-packages/charmhelpers/fetch/ubuntu.py", line 1003, in get_apt_dpkg_env
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install     'PATH': get_system_env('PATH', '/usr/sbin:/usr/bin:/sbin:/bin')}
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install   File "/var/lib/juju/agents/unit-ntp-0/.venv/lib/python3.10/site-packages/charmhelpers/core/host.py", line 1295, in get_system_env
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install     output = subprocess.check_output(
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install   File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install   File "/usr/lib/python3.10/subprocess.py", line 526, in run
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install     raise CalledProcessError(retcode, process.args,
unit-ntp-0: 18:28:11 WARNING unit.ntp/0.install subprocess.CalledProcessError: Command '['env', '-i', '/bin/bash', '-c', 'set -a && source /etc/environment && env']' returned non-zero exit status 127.
unit-ntp-0: 18:28:11 ERROR juju.worker.uniter.operation hook "install" (via explicit, bespoke hook script) failed: exit status 1
unit-ntp-0: 18:28:11 INFO juju.worker.uniter awaiting error resolution for "install" hook

Versions

Operating system: Ubuntu 22.04
Juju : 3.6
Charm revision: Mongodb 6/stable 199

Additional context

Affecting the handover of a customer deployment

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-7295.

This message was autogenerated

[gustavosr98]

Workaround with Juju

juju wait-for application -m $MODEL ntp --timeout 3600s --query='name=="ntp" && (status=="error")'
juju exec -m $MODEL --app ntp -- sudo sed -i 's|^MONGOD_ARGS=\(.*\)$|MONGOD_ARGS="\1"|' /etc/environment

Workaround with Terraform

resource "terraform_data" "juju_wait_for_s3_integrator_blocked" {
  depends_on = [juju_integration.ntp_mongodb]

  provisioner "local-exec" {
    command = <<-EOT
      echo "Workaround for https://github.com/canonical/mongodb-operator/issues/555"
      echo "WARN: Remove if issue fixed"
      juju wait-for application -m $MODEL ntp --timeout 3600s --query='name=="ntp" && (status=="error")'
      juju exec -m $MODEL --app ntp -- sudo sed -i 's|^MONGOD_ARGS=\(.*\)$|MONGOD_ARGS="\1"|' /etc/environment
    EOT
    environment = {
      MODEL = var.model_name
    }
  }
}

[Gu1nness]

@gustavosr98 With the incoming stable that should be released in a few days, this will not be an issue anymore.
We've migrated the config to use a config file instead of /etc/environment/ so this issue will disappear.
I'll close it when the charm is released.