AutoKuma not adding monitors with Service labels and Swarm #116
Description
I deployed AutoKuma to my Docker Swarm and it wasn't creating any monitors in Uptime Kuma. It wouldn't even add a test static monitor I had configured. After enabling trace logging, I was able to see that it was logging into Uptime Kuma successfully, but wasn't doing anything else. There are no errors in the log file.
After troubleshooting this for some time, I eventually tried removing the AUTOKUMA__DOCKER__SOURCE variable, and it started adding monitors via the container labels. Manually setting AUTOKUMA__DOCKER__SOURCE to Services or Both seems to break AutoKuma.
I also tried deploying it on a standalone Docker host. Setting AUTOKUMA__DOCKER__SOURCE to Services or Both causes similar behavior. Removing the variable, or setting it to 'Containers', allows AutoKuma to work correctly.
Note: This was done against a fresh install of Uptime Kuma v1 (initial testing with v2 showed similar results)
Stack YAML:
version: "3.7"
services:
uptime-kuma:
image: louislam/uptime-kuma:1
environment:
PUID: 1001
PGID: 1001
TZ: America/Chicago
UPTIME_KUMA_CLOUDFLARED_TOKEN: "{{ app.uptime_kuma.cloudflared.token }}"
UPTIME_KUMA_DB_TYPE: "mariadb"
UPTIME_KUMA_DB_HOSTNAME: "{{ app.uptime_kuma.db.hostname }}"
UPTIME_KUMA_DB_NAME: "{{ app.uptime_kuma.db.name }}"
UPTIME_KUMA_DB_USERNAME: "{{ app.uptime_kuma.db.username }}"
UPTIME_KUMA_DB_PASSWORD: "{{ app.uptime_kuma.db.password }}"
volumes:
- "/home/synohomes/docker/uptime-kuma/data:/app/data"
ports:
- "3001:3001"
deploy:
labels:
shepard_autoupdate: "true"
networks:
plexreq_backend:
aliases:
- uptime-kuma
socket_proxy_net:
autokuma:
image: ghcr.io/bigboot/autokuma:master
depends_on:
- uptime-kuma
environment:
AUTOKUMA__KUMA__URL: http://uptime-kuma:3001
AUTOKUMA__KUMA__USERNAME: {{ app.autokuma.kuma.username }}
AUTOKUMA__KUMA__PASSWORD: {{ app.autokuma.kuma.password }}
AUTOKUMA__TAG_NAME: AutoKuma
AUTOKUMA__SYNC_INTERVAL: "30.0"
AUTOKUMA__STATIC_MONITORS: "/data/static"
AUTOKUMA__ON_DELETE: delete
AUTOKUMA__DOCKER__HOSTS: "tcp://socket-proxy-dokpv01:2375;tcp://socket-proxy-dokpv02:2375;tcp://socket-proxy-dokpv03:2375"
AUTOKUMA__DOCKER__LABEL_PREFIX: kuma
AUTOKUMA__DOCKER__SOURCE: Both
# Enable the below for debugging purposes
RUST_LOG: "kuma_client=trace"
AUTOKUMA__LOG_DIR: "/data/logs"
volumes:
- "/home/synohomes/docker/autokuma/data:/data"
- "/home/synohomes/docker/autokuma/config/autokuma.toml:/autokuma.toml:ro"
deploy:
placement:
constraints:
- "node.role==manager"
networks:
plexreq_backend:
aliases:
- autokuma
socket_proxy_net:
networks:
plexreq_backend:
external: true
socket_proxy_net:
external: true
Autokuma docker log:
.:::. .:::.
.===-====: :-===--==:
.==. .:==-. .......... :==-. .==:
-=- :===--====================---==: -==
-=- :===-.. ..:===- :==
-=- :: .-. -==
:== ==-
==. .==.
:==- -==-
.====. ====-
==- .==:
:== ===
-== -==
-== :==
-== .. ... .. -==
.==. :=== -=====. ==== ==-
=== .:. :==- :==- :: :==.
.==: :==: .==- .==:
.==. :==: .==- .==-
.==: .==: .==: .==:
.==- ==- :== :==:
.-==: :== -=- .-==.
.===. ==. .::::.. ==. .-==:
:===. :=- ==========. :== .-==:
.===: ==. -========= .==. .-==:
.-==-:== .======: ==-:===:
:-===: ... .====:.
:==-. .-==:
:====---:--====:
.::----::.
_ _ __
/\ | | | |/ /
/ \ _ _ | |_ ___ | ' / _ _ _ __ ___ __ _
/ /\ \ | | | || __| / _ \ | < | | | || '_ ` _ \ / _` |
/ ____ \ | |_| || |_ | (_) || . \ | |_| || | | | | || (_| |
/_/ \_\ \__,_| \__| \___/ |_|\_\ \__,_||_| |_| |_| \__,_|
v0.8.0-3236fb93
DEBUG [kuma_client::client] Waiting for connection
DEBUG [kuma_client::client] Connection opened!
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
TRACE [kuma_client::client] Client::on_any(Custom("info"), Text([Object {"primaryBaseURL": Null, "serverTimezone": String("America/Chicago"), "serverTimezoneOffset": String("-06:00")}]))
DEBUG [kuma_client::client] call login -> Text([Array [Object {"ok": Bool(true), "token": String("eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6Imt1bWFhZG1pbiIsImgiOiJiZDgwZDdjYzhhOTI3ZTJlOGQ1ZTIxYTgzZTY0YjVlNyIsImlhdCI6MTczODUwNTQ2OH0.ANI0aaMGukUl6-XmGtmpSkdBk52rOoA8ZG1Z8jCs2Ow")}]])
DEBUG [kuma_client::client] Logged in as kumaadmin!
TRACE [kuma_client::client] Client::on_any(Custom("monitorList"), Text([Object {}]))
TRACE [kuma_client::client] Client::on_any(Custom("maintenanceList"), Text([Object {}]))
TRACE [kuma_client::client] Client::on_any(Custom("info"), Text([Object {"isContainer": Bool(true), "latestVersion": String("1.23.15"), "primaryBaseURL": Null, "serverTimezone": String("America/Chicago"), "serverTimezoneOffset": String("-06:00"), "version": String("1.23.16")}]))
TRACE [kuma_client::client] Client::on_any(Custom("notificationList"), Text([Array []]))
TRACE [kuma_client::client] Client::on_any(Custom("proxyList"), Text([Array []]))
TRACE [kuma_client::client] Client::on_any(Custom("dockerHostList"), Text([Array []]))
TRACE [kuma_client::client] Client::on_any(Custom("apiKeyList"), Text([Array []]))
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
TRACE [kuma_client::client] Client::on_any(Custom("statusPageList"), Text([Object {}]))
DEBUG [kuma_client::client] Connected!
DEBUG [kuma_client::client] call getTags -> Text([Array [Object {"ok": Bool(true), "tags": Array []}]])
DEBUG [kuma_client::client] call getTags -> Text([Array [Object {"ok": Bool(true), "tags": Array []}]])
DEBUG [kuma_client::client] Connection closed!
DEBUG [kuma_client::client] Waiting for connection
DEBUG [kuma_client::client] Connection opened!
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
TRACE [kuma_client::client] Client::on_any(Custom("info"), Text([Object {"primaryBaseURL": Null, "serverTimezone": String("America/Chicago"), "serverTimezoneOffset": String("-06:00")}]))
DEBUG [kuma_client::client] call login -> Text([Array [Object {"ok": Bool(true), "token": String("eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6Imt1bWFhZG1pbiIsImgiOiJiZDgwZDdjYzhhOTI3ZTJlOGQ1ZTIxYTgzZTY0YjVlNyIsImlhdCI6MTczODUwNTUwMH0.WkrB14D-AEocIyXOVhNWyabVwB7Hp9pp5Vp4_oofxtU")}]])
DEBUG [kuma_client::client] Logged in as kumaadmin!
TRACE [kuma_client::client] Client::on_any(Custom("monitorList"), Text([Object {}]))
TRACE [kuma_client::client] Client::on_any(Custom("maintenanceList"), Text([Object {}]))
TRACE [kuma_client::client] Client::on_any(Custom("info"), Text([Object {"isContainer": Bool(true), "latestVersion": String("1.23.15"), "primaryBaseURL": Null, "serverTimezone": String("America/Chicago"), "serverTimezoneOffset": String("-06:00"), "version": String("1.23.16")}]))
TRACE [kuma_client::client] Client::on_any(Custom("notificationList"), Text([Array []]))
TRACE [kuma_client::client] Client::on_any(Custom("proxyList"), Text([Array []]))
TRACE [kuma_client::client] Client::on_any(Custom("dockerHostList"), Text([Array []]))
TRACE [kuma_client::client] Client::on_any(Custom("apiKeyList"), Text([Array []]))
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
TRACE [kuma_client::client] Client::on_any(Custom("statusPageList"), Text([Object {}]))
DEBUG [kuma_client::client] Connected!
DEBUG [kuma_client::client] call getTags -> Text([Array [Object {"ok": Bool(true), "tags": Array []}]])
DEBUG [kuma_client::client] call getTags -> Text([Array [Object {"ok": Bool(true), "tags": Array []}]])
DEBUG [kuma_client::client] Connection closed!
DEBUG [kuma_client::client] Waiting for connection
DEBUG [kuma_client::client] Connection opened!
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
TRACE [kuma_client::client] Client::on_any(Custom("info"), Text([Object {"primaryBaseURL": Null, "serverTimezone": String("America/Chicago"), "serverTimezoneOffset": String("-06:00")}]))
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
DEBUG [kuma_client::client] call login -> Text([Array [Object {"ok": Bool(true), "token": String("eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6Imt1bWFhZG1pbiIsImgiOiJiZDgwZDdjYzhhOTI3ZTJlOGQ1ZTIxYTgzZTY0YjVlNyIsImlhdCI6MTczODUwNTUzMX0.dSaRwAnh_yHgnE6iSJEWkrX7OjExeRBfPOktxYu85zw")}]])
DEBUG [kuma_client::client] Logged in as kumaadmin!
TRACE [kuma_client::client] Client::on_any(Custom("monitorList"), Text([Object {}]))
TRACE [kuma_client::client] Client::on_any(Custom("maintenanceList"), Text([Object {}]))
TRACE [kuma_client::client] Client::on_any(Custom("info"), Text([Object {"isContainer": Bool(true), "latestVersion": String("1.23.15"), "primaryBaseURL": Null, "serverTimezone": String("America/Chicago"), "serverTimezoneOffset": String("-06:00"), "version": String("1.23.16")}]))
TRACE [kuma_client::client] Client::on_any(Custom("notificationList"), Text([Array []]))
TRACE [kuma_client::client] Client::on_any(Custom("proxyList"), Text([Array []]))
TRACE [kuma_client::client] Client::on_any(Custom("dockerHostList"), Text([Array []]))
TRACE [kuma_client::client] Client::on_any(Custom("apiKeyList"), Text([Array []]))
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
DEBUG [kuma_client::client] Waiting for Kuma to get ready...
TRACE [kuma_client::client] Client::on_any(Custom("statusPageList"), Text([Object {}]))
DEBUG [kuma_client::client] Connected!
DEBUG [kuma_client::client] call getTags -> Text([Array [Object {"ok": Bool(true), "tags": Array []}]])
DEBUG [kuma_client::client] call getTags -> Text([Array [Object {"ok": Bool(true), "tags": Array []}]])
DEBUG [kuma_client::client] Connection closed!
The configured autokuma.toml file is empty.
Docker version:
Docker version 24.0.2, build cb74dfc
It appears that AutoKuma fails at some point when trying to read the Service labels for some unknown reason. At which point it won't add any monitors, whether configured via static files or container labels.
Could you please add some additional logging to help with troubleshooting issues like this? AutoKuma doesn't seem to log anything about what it was trying to do, what labels it is interpreting, whether it can connect to the Docker sockets, whether the snippet I put in there was understood (I eventually just removed it), etc. The only logs I ever saw were about its connection to Uptime Kuma which, while useful, weren't helpful during the hours of troubleshooting.
Please let me know if you need more info or would like me to test something, I would be happy to help!