Autoscaling (HPA) failed to get CPU consumption: c

Question

I am trying to test the HPA (horizontal pod autoscaling) in my Kubernetes cluster. Heapster is up and running and I think it works well since I'm able to see metrics in Grafana. Also the DNS addon is working perfectly.

Looking inside the HPA I can see the error "failed to get CPU consumption and request: failed to unmarshall heapster response: json: cannot unmarshal object into Go value of type []v1alpha1.PodMetrics"

$ kubectl describe hpa php-apache
Name:               php-apache
Namespace:          default
Labels:             <none>
Annotations:            <none>
CreationTimestamp:      Wed, 10 Aug 2016 13:01:47 +0200
Reference:          Deployment/php-apache
Target CPU utilization:     50%
Current CPU utilization:    <unset>
Min replicas:           1
Max replicas:           10
Events:
  FirstSeen LastSeen    Count   From                SubobjectPath   Type        Reason          Message
  --------- --------    -----   ----                -------------   --------    ------          -------
  36s       6s      3   {horizontal-pod-autoscaler }            Warning     FailedGetMetrics    failed to get CPU consumption and request: failed to unmarshall heapster response: json: cannot unmarshal object into Go value of type []v1alpha1.PodMetrics
  36s       6s      3   {horizontal-pod-autoscaler }            Warning     FailedComputeReplicas   failed to get CPU utilization: failed to get CPU consumption and request: failed to unmarshall heapster response: json: cannot unmarshal object into Go value of type []v1alpha1.PodMetrics

The logs from Heapster says:

$ kubectl logs --namespace=kube-system heapster-vf9h9 -c heapster
I0810 06:38:17.178683       1 heapster.go:66] /heapster --source=kubernetes:https://kubernetes.default --sink=influxdb:http://monitoring-influxdb:8086
I0810 06:38:17.178947       1 heapster.go:67] Heapster version 1.2.0-beta.0
I0810 06:38:17.179091       1 configs.go:60] Using Kubernetes client with master "https://kubernetes.default" and version v1
I0810 06:38:17.179100       1 configs.go:61] Using kubelet port 10255
E0810 06:40:24.467817       1 influxdb.go:217] issues while creating an InfluxDB sink: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp 192.168.112.71:8086: getsockopt: connection timed out, will retry on use
I0810 06:40:24.467853       1 influxdb.go:231] created influxdb sink with options: host:monitoring-influxdb:8086 user:root db:k8s
I0810 06:40:24.467883       1 heapster.go:96] Starting with InfluxDB Sink
I0810 06:40:24.467892       1 heapster.go:96] Starting with Metric Sink
I0810 06:40:24.481233       1 heapster.go:175] Starting heapster on port 8082
I0810 06:41:05.129926       1 influxdb.go:209] Created database "k8s" on influxDB server at "monitoring-influxdb:8086"
I0810 11:02:45.357708       1 handlers.go:190] No metrics for pod default/php-apache-1272773798-rm4i0
I0810 11:02:45.426775       1 handlers.go:190] No metrics for pod default/php-apache-1272773798-rm4i0

My cluster has 2 nodes and the master node is also a node. More info:

$ kubectl cluster-info
Kubernetes master is running at http://10.1.1.46:8080
Heapster is running at http://10.1.1.46:8080/api/v1/proxy/namespaces/kube-system/services/heapster
KubeDNS is running at http://10.1.1.46:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at http://10.1.1.46:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
monitoring-grafana is running at http://10.1.1.46:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0", GitCommit:"283137936a498aed572ee22af6774b6fb6e9fd94", GitTreeState:"clean", BuildDate:"2016-07-01T19:26:38Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.4", GitCommit:"dd6b458ef8dbf24aff55795baa68f83383c9b3a9", GitTreeState:"clean", BuildDate:"2016-08-01T16:38:31Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Is it a problem between the Heapster version and the Kubernetes version? I did the git clone of Heapster today, so it's the most recent release.

Answer 1

It seems that you are using an incompatible version of Heapster. Please try with version 1.1.0, as defined here: https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/cluster-monitoring/influxdb/heapster-controller.yaml