Creating custom scheduler doesn't work

Question

When I follow these instructions to create custom scheduler, the pods assigned to my-scheduler (pod annotation-second-scheduler in the example) keep status Pending and are never scheduled.

I think this is because the kube-scheduler cannot access the master from within the pod. I don't know how to get this working. How can the master be accessed from within a pod? I tried running kubectl proxy -p 8001 in the pod, but this doesn't work.

Answer 1

From the user guide section on accessing the cluster API from a pod at kubernetes.io:

When accessing the API from a pod, locating and authenticating to the api server are somewhat different.

The recommended way to locate the apiserver within the pod is with the kubernetes DNS name, which resolves to a Service IP which in turn will be routed to an apiserver.

The recommended way to authenticate to the apiserver is with a service account credential. By kube-system, a pod is associated with a service account, and a credential (token) for that service account is placed into the filesystem tree of each container in that pod, at /var/run/secrets/kubernetes.io/serviceaccount/token.

If available, a certificate bundle is placed into the filesystem tree of each container at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, and should be used to verify the serving certificate of the apiserver.

Finally, the default namespace to be used for namespaced API operations is placed in a file at /var/run/secrets/kubernetes.io/serviceaccount/namespace in each container.

From within a pod the recommended ways to connect to API are:

• run a kubectl proxy as one of the containers in the pod, or as a background process within a container. This proxies the Kubernetes
  API to the localhost interface of the pod, so that other processes in any container of the pod can access it. See this example of using
  kubectl proxy in a pod.
• use the Go client library, and create a client using the client.NewInCluster() factory. This handles locating and authenticating to the apiserver.

In each case, the credentials of the pod are used to communicate securely with the apiserver.

Answer 2

There are few issues with the instructions mentioned in https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/ for local clusters that I created using the instructions mentioned in https://blog.tekspace.io/setup-kubernetes-cluster-with-ubuntu-16-04/

These errors were reported from custom scheduler container (kubect logs command):

E0628 21:05:29.128618       1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list persistentvolumeclaims at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:29.129945       1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list services at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:29.132968       1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list storageclasses.storage.k8s.io at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:29.151367       1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list persistentvolumes at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:29.152097       1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1beta1.ReplicaSet: replicasets.extensions is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list replicasets.extensions at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:29.153187       1 reflector.go:205] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:594: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list pods at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:29.153201       1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list nodes at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:29.153300       1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list replicationcontrollers at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:29.153338       1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list poddisruptionbudgets.policy at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:29.153757       1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1beta1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list statefulsets.apps at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:30.147954       1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list storageclasses.storage.k8s.io at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:30.149547       1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list persistentvolumeclaims at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found
E0628 21:05:30.149562       1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list services at the cluster scope: clusterrole.rbac.authorization.k8s.io "kube-scheduler" not found

The issue is in the my-scheduler.yaml file: in roleref change the name field from kube-scheduler to system:kube-scheduler. Verify it using this command before changing the yaml file:

 kubectl get clusterrole  --all-namespaces | grep -i kube

It should list system:kube-scheduler instead of kube-scheduler only.

Then, it might print these errors in the custom scheduler container:

E0628 21:22:39.937271       1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list storageclasses.storage.k8s.io at the cluster scope
E0628 21:22:40.940461       1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list storageclasses.storage.k8s.io at the cluster scope
E0628 21:22:41.943323       1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list storageclasses.storage.k8s.io at the cluster scope
E0628 21:22:42.946263       1 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list storageclasses.storage.k8s.io at the cluster scope

In this case, please append these lines:

- apiGroups:
  - storage.k8s.io
  resources:
  - storageclasses
  verbs:
  - watch
  - list
  - get

to the end of the output of this command (this opens a file for you to edit):

kubectl edit clusterrole system:kube-scheduler