Horizontal Pod Autoscaler (HPA) not working on Tectonic 1.9.6 k8s cluster

[parveenk27]

Issue Report Template

Tectonic Version

tectonic_1.9.6-tectonic.1

Environment

AWS

Expected Behavior

Actual Behavior

$ kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10

$ kubectl describe hpa/php-apache
Name:                                                  php-apache
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Fri, 10 Aug 2018 23:46:05 +0530
Reference:                                             Deployment/php-apache
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 50%
Min replicas:                                          1
Max replicas:                                          10
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
Events:
  Type     Reason                   Age                  From                       Message
  ----     ------                   ----                 ----                       -------
  Warning  FailedGetResourceMetric  1m (x1641 over 13h)  horizontal-pod-autoscaler  unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)

Reproduction Steps

Step 1.

$ kubectl run php-apache --image=k8s.gcr.io/hpa-example --requests=cpu=200m --expose --port=80
service "php-apache" created
deployment "php-apache" created

Step 2.

$ kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10
deployment "php-apache" autoscaled

Step 3.

$ kubectl get hpa
NAME         REFERENCE               TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   <unknown> / 50%   1         10        1          16m

Other Information

I've also checked heapster.

$ kubectl get logs heapster-xxxxx-xxx heapster -n kube-system
I0809 07:32:19.698269       1 heapster.go:72] /heapster --source=kubernetes.summary_api:''
I0809 07:32:19.698315       1 heapster.go:73] Heapster version v1.4.1
I0809 07:32:19.698541       1 configs.go:61] Using Kubernetes client with master "https://10.3.0.1:443" and version v1
I0809 07:32:19.698559       1 configs.go:62] Using kubelet port 10255
I0809 07:32:19.699336       1 heapster.go:196] Starting with Metric Sink
I0809 07:32:19.999567       1 heapster.go:106] Starting heapster on port 8082

[parveenk27]

I've found the similar issue : kubernetes/kubernetes#57673

[parveenk27]

I found the solution from above mentioned issue, and I think It's a bug in Tectonic installer config.
Change kube-controller-manager's parameter --horizontal-pod-autoscaler-use-rest-clients to false and It works. Actually the issue is, the default value of parameter --horizontal-pod-autoscaler-use-rest-clients is true in k8s 1.9.x.

[pierrebeaucamp]

Any way of getting this to work without re-deploying the cluster?

[parveenk27]

You could try to edit your cluster's kube-controller-manager Deployment config and rollout the Deployment.

[pierrebeaucamp]

Yes, I thought about that, but would this actually be persistent across updates?

[parveenk27]

I'm not sure, But I think it will stay persistent because after all it's a Deployment config and It will be get stored in etcd.

[pierrebeaucamp]

Oh I'm not worried about that particular scenario - I was thinking of the case where tectonic pushes an update to the hyperkube image. The update agent would need to re-deploy the controller manager using an edited configuration and I'd assume it would therefore overwrite any changes I did to it.

Either way, I'll try adding that config parameter to the deployment - I can always re-add it for now, but I hope that a future update would eliminate the need for it to exist.