Skip to content

amazon-ssm-agent.service keeps going inactive (dead) #637

@ryantepera

Description

@ryantepera

I am running amazon-ssm-agent.service v3.3.2656.0 with the following systemd configurations on a RHEL 8 system

[Unit]
Description=amazon-ssm-agent
After=network-online.target

[Service]
Type=simple
WorkingDirectory=/usr/bin/
ExecStart=/usr/bin/amazon-ssm-agent
KillMode=process

# Restart the agent regardless of whether it crashes (and returns a non-zero result code) or if
# is terminated normally (e.g. via 'kill -HUP').  Delay restart so that the agent is less likely
# to restart during a reboot initiated by a script. If the agent exits with status 194 (reboot
# requested), don't restart at all.
Restart=always
RestartPreventExitStatus=194
RestartSec=90

[Install]
WantedBy=multi-user.target

But it seem my ssm agent consistently runs into the problem where it goes inactive with the following terminated signal

amazon-ssm-agent got signal:terminated value:0x7f292bd0ce20
Jun 25 00:07:49 terratest-regression-qlic7m.dev.ac2sp.army.mil amazon-ssm-agent[1229]: 2025-06-25 00:07:49.9846 INFO [amazon-ssm-agent] Stopping Core Agent
Jun 25 00:07:49 terratest-regression-qlic7m.dev.ac2sp.army.mil amazon-ssm-agent[1229]: 2025-06-25 00:07:49.9848 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] Receiving stop signal, stop worker monitor
Jun 25 00:07:56 terratest-regression-qlic7m.dev.ac2sp.army.mil amazon-ssm-agent[1229]: 2025-06-25 00:07:56.9886 INFO [CredentialRefresher] Sending credential refresher stop signal
Jun 25 00:07:56 terratest-regression-qlic7m.dev.ac2sp.army.mil amazon-ssm-agent[1229]: 2025-06-25 00:07:56.9886 INFO [Registrar] Registrar is already stopped
Jun 25 00:07:56 terratest-regression-qlic7m.dev.ac2sp.army.mil amazon-ssm-agent[1229]: 2025-06-25 00:07:56.9888 INFO [CredentialRefresher] Stopping credentials refresher
Jun 25 00:07:56 terratest-regression-qlic7m.dev.ac2sp.army.mil amazon-ssm-agent[1229]: 2025-06-25 00:07:56.9888 INFO [amazon-ssm-agent] Bye.
Jun 25 00:07:56 terratest-regression-qlic7m.dev.ac2sp.army.mil systemd[1]: amazon-ssm-agent.service: Succeeded.
Jun 25 00:07:56 terratest-regression-qlic7m.dev.ac2sp.army.mil systemd[1]: Stopped amazon-ssm-agent.

I'm not sure what value:0x7f292bd0ce20 indicates and was hoping someone could answer that for me.

Here is a full journalctl -u log:

-- Logs begin at Tue 2025-06-24 23:37:07 UTC, end at Tue 2025-06-24 23:39:53 UTC. --
Jun 24 23:37:35 ip-10-244-13-209.dev.ac2sp.army.mil systemd[1]: Started amazon-ssm-agent.
Jun 24 23:37:35 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: Error occurred fetching the seelog config file path:  open /etc/amazon/ssm/seelog.xml: no such file or directory
Jun 24 23:37:35 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: Initializing new seelog logger
Jun 24 23:37:35 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: New Seelog Logger Creation Complete
Jun 24 23:37:35 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:35.7201 INFO Proxy environment variables:
Jun 24 23:37:35 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:35.7206 INFO https_proxy:
Jun 24 23:37:35 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:35.7206 INFO http_proxy:
Jun 24 23:37:36 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:35.7206 INFO no_proxy: 169.254.169.254
Jun 24 23:37:36 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:35.7206 INFO Checking if agent identity type OnPrem can be assumed
Jun 24 23:37:36 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:35.7484 INFO Checking if agent identity type EC2 can be assumed
Jun 24 23:37:36 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:36.0551 INFO Agent will take identity from EC2
Jun 24 23:37:36 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:36.0680 INFO [amazon-ssm-agent] amazon-ssm-agent - v3.3.2471.0
Jun 24 23:37:36 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:36.0680 INFO [amazon-ssm-agent] OS: linux, Arch: amd64
Jun 24 23:37:36 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:36.0680 INFO [amazon-ssm-agent] Starting Core Agent
Jun 24 23:37:36 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:36.0680 INFO [amazon-ssm-agent] Registrar detected. Attempting registration
Jun 24 23:37:36 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:36.0680 INFO [Registrar] Starting registrar module
Jun 24 23:37:36 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:36.1052 INFO [EC2Identity] Checking disk for registration info
Jun 24 23:37:36 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:36.1087 INFO [EC2Identity] No registration info found for ec2 instance, attempting registration
Jun 24 23:37:36 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:36.1087 INFO [EC2Identity] Found registration keys
Jun 24 23:37:36 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:36.1087 INFO [EC2Identity] Checking write access before registering
Jun 24 23:37:36 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:36.1122 INFO [EC2Identity] Registering EC2 instance with Systems Manager
Jun 24 23:37:36 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:36.1772 INFO [EC2Identity] EC2 registration was successful.
Jun 24 23:37:36 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:36.1773 INFO [amazon-ssm-agent] Registration attempted. Resuming core agent startup.
Jun 24 23:37:36 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:36.1779 INFO [CredentialRefresher] credentialRefresher has started
Jun 24 23:37:36 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:36.1779 INFO [CredentialRefresher] Starting credentials refresher loop
Jun 24 23:37:36 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:36.2074 INFO EC2RoleProvider Successfully connected with instance profile role credentials
Jun 24 23:37:36 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:36.2106 INFO [CredentialRefresher] Credentials ready
Jun 24 23:37:36 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:36.2111 INFO [CredentialRefresher] Next credential rotation will be in 29.999939193016665 minutes
Jun 24 23:37:37 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:37.4651 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] [WorkerProvider] Worker ssm-agent-worker is not running, starting worker process
Jun 24 23:37:37 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:37.4770 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] [WorkerProvider] Worker ssm-agent-worker (pid:1265) started
Jun 24 23:37:37 ip-10-244-13-209.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:37:37.4771 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] Monitor long running worker health every 60 seconds
Jun 24 23:37:59 terratest-regression-ku6kkq.dev.ac2sp.army.mil useradd[1789]: new group: name=ssm-user, GID=1006
Jun 24 23:37:59 terratest-regression-ku6kkq.dev.ac2sp.army.mil useradd[1789]: new user: name=ssm-user, UID=1006, GID=1006, home=/home/ssm-user, shell=/bin/bash
Jun 24 23:38:04 terratest-regression-ku6kkq.dev.ac2sp.army.mil sudo[1822]: ssm-user : TTY=pts/0 ; PWD=/usr/bin ; USER=root ; COMMAND=/bin/su
Jun 24 23:38:14 terratest-regression-ku6kkq.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:38:14.8953 INFO [amazon-ssm-agent] amazon-ssm-agent got signal:terminated value:0x7fc241c2fe20
Jun 24 23:38:14 terratest-regression-ku6kkq.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:38:14.8954 INFO [amazon-ssm-agent] Stopping Core Agent
Jun 24 23:38:14 terratest-regression-ku6kkq.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:38:14.8955 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] Receiving stop signal, stop worker monitor
Jun 24 23:38:14 terratest-regression-ku6kkq.dev.ac2sp.army.mil systemd[1]: Stopping amazon-ssm-agent...
Jun 24 23:38:21 terratest-regression-ku6kkq.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:38:21.9191 INFO [CredentialRefresher] Sending credential refresher stop signal
Jun 24 23:38:21 terratest-regression-ku6kkq.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:38:21.9191 INFO [Registrar] Registrar is already stopped
Jun 24 23:38:21 terratest-regression-ku6kkq.dev.ac2sp.army.mil amazon-ssm-agent[1216]: 2025-06-24 23:38:21.9192 INFO [amazon-ssm-agent] Bye.
Jun 24 23:38:22 terratest-regression-ku6kkq.dev.ac2sp.army.mil systemd[1]: amazon-ssm-agent.service: Succeeded.
Jun 24 23:38:22 terratest-regression-ku6kkq.dev.ac2sp.army.mil systemd[1]: Stopped amazon-ssm-agent.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions