tectonic-etcd-operator pod in state ImagePullBackOff

[RonnyMaas]

Issue Report Template

Tectonic Version

tectonic_1.9.6-tectonic.1

Environment

What hardware/cloud provider/hypervisor is being used with Tectonic?
vmware/vsphere

Expected Behavior

Running tectonic-etcd-operator pod

Actual Behavior

Only tectonic-etcd-operator pod is in state ImagePullBackOff. Reason "Permission Denied" when pulling image.

All other pods are running and Cluster appears to be fully functional.

Failed to pull image "quay.io/coreos/tectonic-etcd-operator:v0.0.2": rpc error: code = Unknown desc = Error: Status 403 trying to pull repository coreos/tectonic-etcd-operator: "{"error": "Permission Denied (authorized: robot coreos+tec2_ifbugljtgbbden2bgzcs2nrui)"}"

Reproduction Steps

1. Clean install new tectonic_1.9.6-tectonic.1 cluster

Other Information

Feature Request

Environment

What hardware/cloud provider/hypervisor is being used with Tectonic?

Desired Feature

Other Information

[RonnyMaas]

I didn't have this issue with tectonic_1.8.9-tectonic.2

[RonnyMaas]

Also have this in my logging. But 1.8.9 is using same version (tectonic-etcd-operator:v0.0.2)

May 29 09:43:47 worker-02 env[3264]: time="2018-05-29T09:43:47.864773469Z" level=error msg="Handler for GET /v1.27/images/quay.io/coreos/tectonic-etcd-operator:v0.0.2/json returned error: No such image: quay.io/coreos/tectonic-etcd-operator:v0.0.2"

[robszumski]

@RonnyMaas Are you using this with self-hosted etcd? Any more details?

[RonnyMaas]

yes I use this with self-hosted etcd and behind a corporate proxy.

exact same config with 1.8.9-tectonic.2 didn't have this issue.

[RonnyMaas]

logs on masters only show the status for Pending pod, and on the worker where the pod is pendig only the same logs as mentioned above and this one below are repeated on and on.

May 29 20:42:55 worker-03 kubelet-wrapper[3472]: I0529 20:42:55.686898 3472 kuberuntime_manager.go:514] Container {Name:tectonic-etcd-operator Image:quay.io/coreos/tectonic-etcd-operator:v0.0.2 Command:[/usr/local/bin/tectonic-etcd-operator] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[cpu:{i:{value:100 scale:-3} d:{Dec:} s:100m Format:DecimalSI} memory:{i:{value:134217728 scale:0} d:{Dec:} s: Format:BinarySI}] Requests:map[cpu:{i:{value:20 scale:-3} d:{Dec:} s:20m Format:DecimalSI} memory:{i:{value:52428800 scale:0} d:{Dec:} s:50Mi Format:BinarySI}]} VolumeMounts:[{Name:default-token-2tzjn ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
May 29 20:42:55 worker-03 env[3265]: time="2018-05-29T20:42:55.687765611Z" level=error msg="Handler for GET /v1.27/images/quay.io/coreos/tectonic-etcd-operator:v0.0.2/json returned error: No such image: quay.io/coreos/tectonic-etcd-operator:v0.0.2"

[RonnyMaas]

May 29 22:02:43 worker-03 kubelet-wrapper[3472]: E0529 22:02:43.684645 3472 pod_workers.go:186] Error syncing pod 479f2702-6380-11e8-ae93-005056a02949 ("tectonic-etcd-operator-84c7dd9684-mllb5_tectonic-system(479f2702-6380-11e8-ae93-005056a02949)"), skipping: failed to "StartContainer" for "tectonic-etcd-operator" with ImagePullBackOff: "Back-off pulling image "quay.io/coreos/tectonic-etcd-operator:v0.0.2""

[RonnyMaas]

Should tectonic-etcd-operator even try to start/run on a vmware installer deployment?

I just upgraded a 1.8.4 cluster (bare-metal installer) also with self-hosted etcd but that one doesn't have a tectonic-etcd-operator deployment. It only has etcd-operator.

The vsphere deployment has both the tectonic-etcd-operator and etcd-operator.