feat: COS certificates integrations (e-2-e TLS)

[MichaelThamm]

Related to:

• feat: COS Lite TF module TLS integrations #40

Drive by PR:

• fix: ingress databag when TLS information exists loki-coordinator-k8s-operator#69

Deploy with TLS

Create a observability-stack/terraform/modules/cos/tls.tfvars file:

model                         = "cos"
channel                       = "2/edge"
ssc_channel                   = "1/edge"
traefik_channel               = "latest/edge"
use_tls                       = true
loki_bucket                   = "loki"
mimir_bucket                  = "mimir"
tempo_bucket                  = "tempo"
s3_endpoint                   = "http://REPLACE_ME:8080"  # WITH `ip -4 -j route get 2.2.2.2 | jq -r '.[] | .prefsrc'`
s3_access_key                 = "access-key"
s3_secret_key                 = "secret-key"
loki_backend_units            = 1
loki_read_units               = 1
loki_write_units              = 1
loki_coordinator_units        = 1
mimir_backend_units           = 1
mimir_read_units              = 1
mimir_write_units             = 1
mimir_coordinator_units       = 1
tempo_compactor_units         = 1
tempo_distributor_units       = 1
tempo_ingester_units          = 1
tempo_metrics_generator_units = 1
tempo_querier_units           = 1
tempo_query_frontend_units    = 1
tempo_coordinator_units       = 1

Context

• https://charmhub.io/topics/security-with-x-509-certificates

Testing

Ref: traefik bundles

• Test TLS is working
  • The dashboards had some data (some panels were missing data, likely unrelated).
  • In traefik container:
    • Get each of the ingress kube-dns names for each charm with ls -1 /opt/traefik/juju/ and inspect the YAML files
    • curl https://alertmanager-0.alertmanager-endpoints.cos-cos.svc.cluster.local:9093
    • echo | openssl s_client -showcerts -connect alertmanager-0.alertmanager-endpoints.cos.svc.cluster.local:9093 2>/dev/null | openssl x509 -text | grep -C 5 DNS
    • echo | openssl s_client -strict -verify_return_error -connect alertmanager-0.alertmanager-endpoints.cos.svc.cluster.local:9093 || echo "failed"
      • Repeat for each charm in the model:
        • Mimir
        • Loki
        • Tempo
          • Tested ports: 14250, 14268, 4317, 4318, 9096, 3200, 9411
        • Alertmanager
        • Catalogue
  • From Juju client:
    • juju run self-signed-certificates/0 get-ca-certificate | yq -r '.ca-certificate' > ssc.cert
      • curl -L --fail-with-body --capath $PWD --cacert ssc.cert https://192.168.88.12/cos-alertmanager/#/alerts