Deploy my usual k8s Pulp CR fails due to postgres django version

[jear]

Problem:
following procedure to install through operator
https://pulpproject.org/pulp-operator/docs/admin/guides/install/helm/

All pods in error complains about
django.db.utils.NotSupportedError: PostgreSQL 14 or later is required (found 13.23).
How can I restore my previous install ?

what is the exact version of 'stable' in :
image: quay.io/pulp/pulp-minimal:stable

Please help !
Best

(my-tasks) jear@devops-002:~/workspace/my-pulp$ cat my-pulp.yaml
apiVersion: repo-manager.pulpproject.org/v1
kind: Pulp
metadata:
  name: pulp
  namespace: pulp
spec:
  allowed_content_checksums: ["sha256", "sha384", "sha512"]
  api:
    replicas: 1
  content:
    replicas: 1
  worker:
    replicas: 1
  cache:
    enabled: true
    redis_storage_class: rook-ceph-block
  web:
    replicas: 1
  database:
    postgres_storage_class: rook-ceph-block

#  file_storage_storage_class: my-sc-for-pulpcore
  file_storage_storage_class: rook-cephfs
  file_storage_access_mode: "ReadWriteMany"
  file_storage_size: "750Gi"

(my-tasks) jear@devops-002:~/workspace/my-pulp$ k get pods -n pulp
NAME                                                READY   STATUS             RESTARTS        AGE
pulp-api-dc8df6749-6q52p                            0/1     CrashLoopBackOff   8 (116s ago)    18m
pulp-content-756f6d8b6b-mnwfz                       1/1     Running            0               18m
pulp-database-0                                     1/1     Running            0               18m
pulp-operator-controller-manager-59b867cf8f-8rp76   2/2     Running            0               34m
pulp-pulpcore-migration-l64qq-8f6gg                 0/1     Error              0               18m
pulp-pulpcore-migration-l64qq-fw8nl                 0/1     Error              0               17m
pulp-pulpcore-migration-l64qq-sxq5p                 0/1     Error              0               18m
pulp-redis-78d4d7fdc8-fxw6q                         1/1     Running            0               18m
pulp-reset-admin-password-hxkgk-9kr7p               0/1     Error              0               17m
pulp-reset-admin-password-hxkgk-krnb5               0/1     Error              0               18m
pulp-reset-admin-password-hxkgk-vwf7v               0/1     Error              0               18m
pulp-web-85bbd698c7-fqc5h                           0/1     Running            0               18m
pulp-worker-56cdb4fcf6-nk4t7                        0/1     CrashLoopBackOff   8 (2m24s ago)   18m

Expected outcome:
all pods should be up and running

Pulpcore version:
I have tested both operator 0.4 and 0.5

Pulp plugins installed and their versions:
none yet

Operating system - distribution and version:
kubernetes 1.32 vanilla on Ubuntu 24.04

Other relevant data:

k logs -n pulp pulp-api-dc8df6749-6q52p

[2026-01-14 20:16:29 +0000] [9] [ERROR] Exception in worker process
Traceback (most recent call last):
 File "/usr/local/lib/python3.11/site-packages/gunicorn/arbiter.py", line 608, in spawn_worker
   worker.init_process()
 File "/usr/local/lib/python3.11/site-packages/pulpcore/app/entrypoint.py", line 72, in init_process
   self.app_status = AppStatus.objects.create(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/pulpcore/app/models/status.py", line 33, in create
   obj = super().create(app_type=app_type, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
   return getattr(self.get_queryset(), name)(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 665, in create
   obj.save(force_insert=True, using=self.db)
 File "/usr/lib64/python3.11/contextlib.py", line 80, in inner
   with self._recreate_cm():
 File "/usr/local/lib/python3.11/site-packages/django/db/transaction.py", line 198, in __enter__
   if not connection.get_autocommit():
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 454, in get_autocommit
   self.ensure_connection()
 File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
   return func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 279, in ensure_connection
   self.connect()
 File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
   return func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 258, in connect
   self.init_connection_state()
 File "/usr/local/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 398, in init_connection_state
   super().init_connection_state()
 File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 225, in init_connection_state
   self.check_database_version_supported()
 File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 201, in check_database_version_supported
   raise NotSupportedError(
django.db.utils.NotSupportedError: PostgreSQL 14 or later is required (found 13.23).
[2026-01-14 20:16:29 +0000] [9] [INFO] Worker exiting (pid: 9)
[2026-01-14 20:16:30 +0000] [1] [ERROR] Worker (pid:8) exited with code 3
[2026-01-14 20:16:30 +0000] [1] [ERROR] Worker (pid:9) was sent SIGTERM!
[2026-01-14 20:16:30 +0000] [1] [ERROR] Shutting down: Master
[2026-01-14 20:16:30 +0000] [1] [ERROR] Reason: Worker failed to boot.

 k get pods  -n pulp pulp-api-dc8df6749-n56rv -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/containerID: 64334c36cc336aa55dd494b68cde2ee5be7208e5f9b60352186e5a7364a299c9
    cni.projectcalico.org/podIP: 172.20.34.135/32
    cni.projectcalico.org/podIPs: 172.20.34.135/32
    kubectl.kubernetes.io/default-container: api
  creationTimestamp: "2026-01-14T20:38:39Z"
  generateName: pulp-api-dc8df6749-
  labels:
    app: pulp-api
    app.kubernetes.io/component: api
    app.kubernetes.io/instance: pulp-api-pulp
    app.kubernetes.io/managed-by: pulp-operator
    app.kubernetes.io/name: pulp-api
    app.kubernetes.io/part-of: pulp
    pod-template-hash: dc8df6749
    pulp_cr: pulp
  name: pulp-api-dc8df6749-n56rv
  namespace: pulp
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: pulp-api-dc8df6749
    uid: a3223512-93ef-4675-815a-945d982b553e
  resourceVersion: "134429"
  uid: f5e3aa8f-54bb-416c-9824-e73be00c8f22
spec:
  affinity: {}
  containers:
  - args:
    - -c
    - |-
      if which pulpcore-api
      then
        PULP_API_ENTRYPOINT=("pulpcore-api")
      else
        PULP_API_ENTRYPOINT=("gunicorn" "pulpcore.app.wsgi:application" "--name" "pulp-api" "--access-logformat" "pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s \"%(r)s\" %(s)s %(b)s \"%(f)s\" \"%(a)s\"")
      fi
      exec "${PULP_API_ENTRYPOINT[@]}" \
      --bind "[::]:24817" \
      --timeout "${PULP_GUNICORN_TIMEOUT}" \
      --workers "${PULP_API_WORKERS}" \
      --access-logfile -
    command:
    - /bin/sh
    env:
    - name: PULP_GUNICORN_TIMEOUT
      value: "90"
    - name: PULP_API_WORKERS
      value: "2"
    - name: POSTGRES_SERVICE_HOST
      value: pulp-database-svc
    - name: POSTGRES_SERVICE_PORT
      value: "5432"
    - name: REDIS_SERVICE_HOST
      value: pulp-redis-svc.pulp
    - name: REDIS_SERVICE_PORT
      value: "6379"
    image: quay.io/pulp/pulp-minimal:stable
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 10
      httpGet:
        path: /pulp/api/v3/status/
        port: 24817
        scheme: HTTP
      initialDelaySeconds: 3
      periodSeconds: 20
      successThreshold: 1
      timeoutSeconds: 10
    name: api
    ports:
    - containerPort: 24817
      protocol: TCP
    readinessProbe:
      exec:
        command:
        - /usr/bin/readyz.py
        - /pulp/api/v3/status/
      failureThreshold: 1
      initialDelaySeconds: 3
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 10
    resources: {}
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      runAsNonRoot: true
      seccompProfile:
        type: RuntimeDefault
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/pulp/settings.py
      name: pulp-server
      readOnly: true
      subPath: settings.py
    - mountPath: /etc/pulp/keys/database_fields.symmetric.key
      name: pulp-db-fields-encryption
      readOnly: true
      subPath: database_fields.symmetric.key
    - mountPath: /etc/pulp/pulp-admin-password
      name: pulp-admin-password
      readOnly: true
      subPath: admin-password
    - mountPath: /var/lib/pulp
      name: file-storage
    - mountPath: /etc/pulp/keys/container_auth_private_key.pem
      name: pulp-container-auth-certs
      readOnly: true
      subPath: container_auth_private_key.pem
    - mountPath: /etc/pulp/keys/container_auth_public_key.pem
      name: pulp-container-auth-certs
      readOnly: true
      subPath: container_auth_public_key.pem
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-vp558
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  initContainers:
  - args:
    - -c
    - |-
      mkdir -p /var/lib/pulp/{media,assets,tmp}
      /usr/bin/wait_on_postgres.py
      /usr/bin/wait_on_database_migrations.sh
    command:
    - /bin/sh
    env:
    - name: POSTGRES_SERVICE_HOST
      value: pulp-database-svc
    - name: POSTGRES_SERVICE_PORT
      value: "5432"
    image: quay.io/pulp/pulp-minimal:stable
    imagePullPolicy: IfNotPresent
    name: init-container
    resources: {}
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      runAsNonRoot: true
      seccompProfile:
        type: RuntimeDefault
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/pulp/settings.py
      name: pulp-server
      readOnly: true
      subPath: settings.py
    - mountPath: /etc/pulp/keys/database_fields.symmetric.key
      name: pulp-db-fields-encryption
      readOnly: true
      subPath: database_fields.symmetric.key
    - mountPath: /var/lib/pulp
      name: file-storage
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-vp558
      readOnly: true
  nodeName: ag-mks01-worker-1
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 700
    runAsUser: 700
  serviceAccount: pulp
  serviceAccountName: pulp
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: pulp-server
    secret:
      defaultMode: 420
      items:
      - key: settings.py
        path: settings.py
      secretName: pulp-server
  - name: pulp-db-fields-encryption
    secret:
      defaultMode: 420
      items:
      - key: database_fields.symmetric.key
        path: database_fields.symmetric.key
      secretName: pulp-db-fields-encryption
  - name: pulp-admin-password
    secret:
      defaultMode: 420
      items:
      - key: password
        path: admin-password
      secretName: pulp-admin-password
  - name: file-storage
    persistentVolumeClaim:
      claimName: pulp-file-storage
  - name: pulp-container-auth-certs
    secret:
      defaultMode: 420
      items:
      - key: container_auth_public_key.pem
        path: container_auth_public_key.pem
      - key: container_auth_private_key.pem
        path: container_auth_private_key.pem
      secretName: pulp-container-auth
  - name: kube-api-access-vp558
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2026-01-14T20:38:46Z"
    status: "True"
    type: PodReadyToStartContainers
  - lastProbeTime: null
    lastTransitionTime: "2026-01-14T20:38:50Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2026-01-14T20:38:39Z"
    message: 'containers with unready status: [api]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2026-01-14T20:38:39Z"
    message: 'containers with unready status: [api]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2026-01-14T20:38:38Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://afc59f1e96c2e09eb0cc5d3d0eb6b959f27632dfce4d2904d5714e99be211351
    image: quay.io/pulp/pulp-minimal:stable
    imageID: quay.io/pulp/pulp-minimal@sha256:7819efdd614436469cb53c6bca36fcd80c333cb472da867142b682ee76b11029
    lastState:
      terminated:
        containerID: containerd://afc59f1e96c2e09eb0cc5d3d0eb6b959f27632dfce4d2904d5714e99be211351
        exitCode: 3
        finishedAt: "2026-01-14T20:39:56Z"
        reason: Error
        startedAt: "2026-01-14T20:39:52Z"
    name: api
    ready: false
    restartCount: 3
    started: false
    state:
      waiting:
        message: back-off 40s restarting failed container=api pod=pulp-api-dc8df6749-n56rv_pulp(f5e3aa8f-54bb-416c-9824-e73be00c8f22)
        reason: CrashLoopBackOff
    volumeMounts:
    - mountPath: /etc/pulp/settings.py
      name: pulp-server
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /etc/pulp/keys/database_fields.symmetric.key
      name: pulp-db-fields-encryption
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /etc/pulp/pulp-admin-password
      name: pulp-admin-password
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /var/lib/pulp
      name: file-storage
    - mountPath: /etc/pulp/keys/container_auth_private_key.pem
      name: pulp-container-auth-certs
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /etc/pulp/keys/container_auth_public_key.pem
      name: pulp-container-auth-certs
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-vp558
      readOnly: true
      recursiveReadOnly: Disabled
  hostIP: 10.180.10.164
  hostIPs:
  - ip: 10.180.10.164
  initContainerStatuses:
  - containerID: containerd://ee12a4921829f9db318830ef587f2e04ad34e23e7177aca8ac08c4213a4137ab
    image: quay.io/pulp/pulp-minimal:stable
    imageID: quay.io/pulp/pulp-minimal@sha256:7819efdd614436469cb53c6bca36fcd80c333cb472da867142b682ee76b11029
    lastState: {}
    name: init-container
    ready: true
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: containerd://ee12a4921829f9db318830ef587f2e04ad34e23e7177aca8ac08c4213a4137ab
        exitCode: 0
        finishedAt: "2026-01-14T20:38:50Z"
        reason: Completed
        startedAt: "2026-01-14T20:38:45Z"
    volumeMounts:
    - mountPath: /etc/pulp/settings.py
      name: pulp-server
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /etc/pulp/keys/database_fields.symmetric.key
      name: pulp-db-fields-encryption
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /var/lib/pulp
      name: file-storage
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-vp558
      readOnly: true
      recursiveReadOnly: Disabled
  phase: Running
  podIP: 172.20.34.135
  podIPs:
  - ip: 172.20.34.135
  qosClass: BestEffort
  startTime: "2026-01-14T20:38:39Z"

[pedro-psb]

In Pulp's matrix channel we've found that the operator CI is broken because it nows uses Django 5.2, which requires PG>=14.
See the matrix thread.

Support for Django 5 was introduced in pulpcore==3.100, so you should be fine using a version <3.100. Stable should be 3.101, not 100% sure.
See quay 's page.

@git-hyagi , we should move this to pulp-operator, right?

[git-hyagi]

Support for Django 5 was introduced in pulpcore==3.100, so you should be fine using a version <3.100. Stable should be 3.101, not 100% sure.

To use a different version you can update Pulp CR with something like:

spec:
  image_version: 3.96
  image_web_version: 3.96

[jear]

thx !

in the meam time, I have been able to get the image from Test env through sha256 digest:

sudo /var/lib/rancher/rke2/bin/ctr -n k8s.io   image export pulp-web-stable-backup.tar quay.io/pulp/pulp-web@sha256:5a08441b2562d16d4d7e519019a577ceddedf3ff04f6d784cb1d401a3ba3d84d

sudo /var/lib/rancher/rke2/bin/ctr --address /run/k3s/containerd/containerd.sock --namespace k8s.io image export pulp-minimal-backup-sha.tar quay.io/pulp/pulp-minimal@sha256:880659f31046373e2ed3dffd52beab5f3424f5a4cf1be329a8b1d142cfc85fd9