🐛 [BUG] - Déploiment AWX

[Baran-Aksoy]

Description

Chart concerné : awx-operator v2.19.1 (releases.yaml)

Problème

Le pod awx-operator-controller-manager reste bloqué en CrashLoopBackOff.
L'image du container sidecar kube-rbac-proxy n'existe plus sur gcr.io :

root@cpn:# crictl pull gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0 2>&1 | tail -3
time="2026-03-09T10:32:30Z" level=error msg="validate service connection: validate CRI v1 image API for endpoint "unix:///var/run/dockershim.sock": rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/run/dockershim.sock: connect: no such file or directory""
E0309 10:32:30.755704 1430399 remote_image.go:180] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image "gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0": failed to resolve image: gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0: not found" image="gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0"
time="2026-03-09T10:32:30Z" level=fatal msg="pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0": failed to resolve image: gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0: not found"
root@cpn:#

Cette adresse a été migrée vers registry.k8s.io.
L'image est disponible à la nouvelle adresse :

root@cpn:# crictl pull registry.k8s.io/kubebuilder/kube-rbac-proxy:v0.15.0 2>&1 | tail -2
time="2026-03-09T10:33:56Z" level=error msg="validate service connection: validate CRI v1 image API for endpoint "unix:///var/run/dockershim.sock": rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/run/dockershim.sock: connect: no such file or directory""
Image is up to date for sha256:7ebda747308b686ef6471bd8377085bce9f210baef15ce2a9bb6c5d8ccd773dc
root@cpn:#

L'image est hardcodée dans le template du chart pour l'opérateur, une :

root@cpn:# grep -n "kube-rbac-proxy" /tmp/awx-test/awx-operator/templates/operator-controller/_operator-controller.tpl
33: image: gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0
34: name: kube-rbac-proxy
root@cpn:#

Est-ce une mauvaise manipulation de ma part, ou bien si ce n'est pas le cas, quelle solution peut être appliquée ?

Etapes de reproduction

1. Aller à '...'
2. Cliquer sur '....'
3. Scroller jusqu'à '....'
4. Voir l'erreur

Captures d'écran

![DESCRIPTION](LINK.png)

Logs

Navigateurs

No response

OS

Linux

Version de la console impactée

No response

Définition du fini

• Le correctif est terminé
• Les tests liés à ce correctif ont été ajoutés
• La communication avec les autres équipes impliquées par ce correctif a été faite

[Baran-Aksoy]

Étape — Décrire le nouveau pod

kubectl --kubeconfig /root/.kube/config-cpn.yaml \
  describe pod -n dso-awx -l control-plane=controller-manager

Résultat complet

Name:             awx-operator-controller-manager-5ccb88d675-9zfz8
Namespace:        dso-awx
Priority:         0
Service Account:  awx-operator-controller-manager
Node:             cpn/10.46.193.47
Start Time:       Wed, 11 Mar 2026 09:00:07 +0000
Labels:           control-plane=controller-manager
                  helm.sh/chart=awx-operator
                  pod-template-hash=5ccb88d675
Annotations:      cni.projectcalico.org/containerID: 70f5aeb73a05f9384b4b2e4a9edc77464604383a7f5afd0a559a458de3242eb5
                  cni.projectcalico.org/podIP: 192.168.187.109/32
                  cni.projectcalico.org/podIPs: 192.168.187.109/32
                  kubectl.kubernetes.io/default-container: awx-manager
Status:           Pending
IP:               192.168.187.109
IPs:
  IP:           192.168.187.109
Controlled By:  ReplicaSet/awx-operator-controller-manager-5ccb88d675
Containers:
  kube-rbac-proxy:
    Container ID:
    Image:         gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0
    Image ID:
    Port:          8443/TCP (https)
    Host Port:     0/TCP (https)
    Args:
      --secure-listen-address=0.0.0.0:8443
      --upstream=http://127.0.0.1:8080/
      --logtostderr=true
      --v=0
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:        5m
      memory:     64Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wg2gh (ro)
  awx-manager:
    Container ID:  containerd://ccc592fa9032fd1d61ec1a190b761c21b17cb5cd86ba4d774bc40e6bad9b74df
    Image:         quay.io/ansible/awx-operator:2.19.1
    Image ID:      quay.io/ansible/awx-operator@sha256:7302e0c8e5a79c11275848b36c63c61d8e7487840164c109b7cda9aa93456901
    Port:          <none>
    Host Port:     <none>
    Args:
      --health-probe-bind-address=:6789
      --metrics-bind-address=127.0.0.1:8080
      --leader-elect
      --leader-election-id=awx-operator
    State:          Running
      Started:      Wed, 11 Mar 2026 09:00:08 +0000
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1500m
      memory:  960Mi
    Requests:
      cpu:      50m
      memory:   32Mi
    Liveness:   http-get http://:6789/healthz delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:  http-get http://:6789/readyz delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      ANSIBLE_GATHERING:   explicit
      ANSIBLE_DEBUG_LOGS:  false
      WATCH_NAMESPACE:     dso-awx (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wg2gh (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  kube-api-access-wg2gh:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    Optional:                false
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                           Age              From               Message
  ----     ------                           ----             ----               -------
  Normal   Scheduled                        8s               default-scheduler  Successfully assigned dso-awx/awx-operator-controller-manager-5ccb88d675-9zfz8 to cpn
  Normal   Pulling                          9s               kubelet            spec.containers{kube-rbac-proxy}: Pulling image "gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0"
  Warning  Failed                           8s               kubelet            spec.containers{kube-rbac-proxy}: Failed to pull image "gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0": rpc error: code = NotFound desc = failed to pull and unpack image "gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0": failed to resolve image: gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0: not found
  Warning  Failed                           8s               kubelet            spec.containers{kube-rbac-proxy}: Error: ErrImagePull
  Normal   Pulled                           8s               kubelet            spec.containers{awx-manager}: Container image "quay.io/ansible/awx-operator:2.19.1" already present on machine
  Normal   Created                          8s               kubelet            spec.containers{awx-manager}: Created container: awx-manager
  Normal   Started                          8s               kubelet            spec.containers{awx-manager}: Started container awx-manager
  Warning  FailedToRetrieveImagePullSecret  7s (x3 over 9s)  kubelet            Unable to retrieve some image pull secrets (redhat-operators-pull-secret); attempting to pull the image may not succeed.
  Normal   BackOff                          7s (x2 over 8s)  kubelet            spec.containers{kube-rbac-proxy}: Back-off pulling image "gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0"
  Warning  Failed                           7s (x2 over 8s)  kubelet            spec.containers{kube-rbac-proxy}: Error: ImagePullBackOff

Cause

L'event clé qui confirme le bug :

Warning  Failed  kubelet  Failed to pull image "gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0":
  rpc error: code = NotFound desc = failed to pull and unpack image
  "gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0": failed to resolve image:
  gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0: not found

Les autres problemes nous avons ont actuellement une solution.

Localisation dans le code

Vérifié directement dans le repo source https://github.com/ansible/awx-operator au tag 2.19.1.Le tag est important, on parle de la version deploye la derniere. PS : version deployé pas commit

Fichier : config/default/manager_auth_proxy_patch.yaml

Contenu complet du fichier (source : https://raw.githubusercontent.com/ansible/awx-operator/2.19.1/config/default/manager_auth_proxy_patch.yaml) :

# This patch inject a sidecar container which is a HTTP proxy for the
# controller manager, it performs RBAC authorization against the Kubernetes API using SubjectAccessReviews.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: controller-manager
  namespace: system
spec:
  template:
    spec:
      containers:
      - name: kube-rbac-proxy
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - "ALL"
        image: gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0
        args:
        - "--secure-listen-address=0.0.0.0:8443"
        - "--upstream=http://127.0.0.1:8080/"
        - "--logtostderr=true"
        - "--v=0"
        ports:
        - containerPort: 8443
          protocol: TCP
          name: https
        resources:
          limits:
            cpu: 500m
            memory: 128Mi
          requests:
            cpu: 5m
            memory: 64Mi
      - name: awx-manager
        args:
        - "--health-probe-bind-address=:6789"
        - "--metrics-bind-address=127.0.0.1:8080"
        - "--leader-elect"
        - "--leader-election-id=awx-operator"

Il s'agit d'un patch Kustomize qui injecte kube-rbac-proxy comme sidecar dans le Deployment controller-manager. Ce n'est pas dans values.yaml — il n'existe aucun champ Helm pour surcharger cette image. Le chart Helm (ansible-community/awx-operator-helm) est généré depuis ce repo au build et intègre ce patch tel quel.

Conséquence : impossible de corriger via --set ou un fichier values custom. Toute correction nécessite de modifier le chart ou d'appliquer un patch post-déploiement.

Vérification des registries

Depuis le nœud cpn (ou via incus exec cpn --), tester les deux registries :

# Ancienne adresse
crictl pull gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0


root@cpn:~# crictl pull gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0
WARN[0000] image connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead. 
ERRO[0000] validate service connection: validate CRI v1 image API for endpoint "unix:///var/run/dockershim.sock": rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/run/dockershim.sock: connect: no such file or directory" 
E0311 09:26:42.905921 3230557 remote_image.go:180] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image \"gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0\": failed to resolve image: gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0: not found" image="gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0"
FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0": failed to resolve image: gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0: not found 
root@cpn:~# 


# Nouvelle adresse
crictl pull registry.k8s.io/kubebuilder/kube-rbac-proxy:v0.15.0


root@cpn:~# crictl pull registry.k8s.io/kubebuilder/kube-rbac-proxy:v0.15.0
WARN[0000] image connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead. 
ERRO[0000] validate service connection: validate CRI v1 image API for endpoint "unix:///var/run/dockershim.sock": rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/run/dockershim.sock: connect: no such file or directory" 
Image is up to date for sha256:7ebda747308b686ef6471bd8377085bce9f210baef15ce2a9bb6c5d8ccd773dc
root@cpn:~# 

[Baran-Aksoy]

Quelles solutions préconisez-vous ?

[Falltrades]

Le passage à la version 3.2.1 devrait résoudre le problème : https://github.com/ansible-community/awx-operator-helm/releases/tag/awx-operator-3.2.1. Tu peux tester et faire une PR dans ce sens.