When the COS charm workloads encounter issues or break for any reason, the Juju status of the charms still shows the workloads as healthy. This is misleading, and can be fixed by adding Pebble readiness and liveness checks to all COS charm workloads, reflecting their status in Juju status.
For example, Grafana shows healthy in Juju status in my cluster while the workload is erroring:
kubectl -n my-cos-model exec -ti grafana-0 -c grafana -- pebble services
Service Startup Current Since
grafana enabled backoff today at 10:58 UTC
kubectl -n my-cos-model exec -ti grafana-0 -c grafana -- pebble checks
Plan has no health checks.

When the COS charm workloads encounter issues or break for any reason, the Juju status of the charms still shows the workloads as healthy. This is misleading, and can be fixed by adding Pebble readiness and liveness checks to all COS charm workloads, reflecting their status in Juju status.
For example, Grafana shows healthy in Juju status in my cluster while the workload is erroring: