kubernetes error : skipping pod synchronization

Question

i have configured kubernetes master on centos 7 and kubernetes node on another node centos 7

services running on kube master:

• kube-controller-manager
• kube-apiserver
• kube-scheduler
• etcd
• flanneld

service running on kube node:

• flanneld
• docker
• kube-proxy
• kubelet

all services are up and running and i could see the api url successfully getting all endpoints. http://kube-master:8080 however, when i am running command kube get nodes , getting following error :

skipping pod synchronization. container runtime is down

I am not getting what this error means and how to resolve this. Please suggest.

Answer 1

I've seen this problem when docker got into some broken state where it was not able to remove a (specific) stopped container and was leaking zombie processes. Had to power-cycle the node in the end.

CentoOS 7 here too, still at Kubernetes 1.10.0 and Docker CE 18.03.

Answer 2

kubelet is the only component with a dependency on the container runtime (Docker in your case). If I were you I'd start investigating the kubelet logs and search for references to Docker. Maybe the user configured to run kubelet doesn't have the necessary permissions to interact with the Docker socket at /var/run/docker.sock.

The content of your logs may help if you need further help.