kubelet failed with kubelet cgroup driver: “cgroup

Question

Configuration for cgroup drive is right in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"

I also checked the Environment with cli

$ systemctl show --property=Environment kubelet | cat Environment=KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf\x20--require-kubeconfig=true KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests\x20--allow-privileged=true KUBELET_NETWORK_ARGS=--network-plugin=cni\x20--cni-conf-dir=/etc/cni/net.d\x20--cni-bin-dir=/opt/cni/bin KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10\x20--cluster-domain=cluster.local KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook\x20--client-ca-file=/etc/kubernetes/pki/ca.crt KUBELET_CADVISOR_ARGS=--cadvisor-port=0 KUBELET_CGROUP_ARGS=--cgroup-driver=systemd

How to reproduce it:

• yum install -y docker-1.12.6
• systemctl enable docker && systemctl start docker
• setenforce 0
• yum install -y kubelet kubeadm
• systemctl enable kubelet && systemctl start kubelet
• systemctl daemon-reload
• systemctl restart kubelet
• kubelet log

Environment:

• Kubernetes version (use kubectl version): 1.7.3
• Cloud provider or hardware configuration**: 4 core 16G RAM
• OS (e.g. from /etc/os-release): CentOS Linux 7 (Core)
• Kernel (e.g. uname -a): Linux 10-8-108-92 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 23 17:05:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: kubeadm

Answer 1

Possible cause

kubelet 1.7.3 not reading config file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf #50748

Solution

Troubleshooting kubeadm

If you are using CentOS and encounter difficulty while setting up the master node， verify that your Docker cgroup driver matches the kubelet config:

docker info | grep -i cgroup
cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

If the Docker cgroup driver and the kubelet config don’t match, change the kubelet config to match the Docker cgroup driver. The flag you need to change is --cgroup-driver. If it’s already set, you can update like so:

sed -i "s/cgroup-driver=systemd/cgroup-driver=cgroupfs/g /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

This can be replaced with:

CG=$(sudo docker info 2>/dev/null | sed -n 's/Cgroup Driver: \(.*\)/\1/p')
sed -i "s/cgroup-driver=systemd/cgroup-driver=$CG/g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Answer 2

Edit this file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf change systemd to cgroupfs". then restart the kubelet systemctl restart kubelet.

Answer 3

It looks like kubelet process did not load the right settings from the /etc/systemd/system/kubelet.service.d/10-kubeadm.conf telling from the error message.

After getting more info from the chat, I think there are several possible ways to go:

1. Switch both kubelet and docker cgroup driver to cgroupfs Download docker from the repo below which uses cgroupfs by default.

   [dockerrepo] 
   name=Docker Repository 
   baseurl=https://yum.dockerproject.org/repo/main/centos/7 
   enabled=1 
   gpgcheck=1 
   gpgkey=https://yum.dockerproject.org/gpg
   

   And change the cgroup driver in kubelet conf as well. Check whether the error happens again and what kubelet loads from its conf.

2. Add more logs in kubelet code to debug it

   This is the logic kubelet uses to get conf from both sides

Answer 4

Install the docker first:

yum install -y https://download.docker.com/linux/centos/7/x86_64/stable/Packages/docker-ce-selinux-17.03.0.ce-1.el7.centos.noarch.rpm

yum install -y https://download.docker.com/linux/centos/7/x86_64/stable/Packages/docker-ce-17.03.0.ce-1.el7.centos.x86_64.rpm

[root@cgfs-dock ~]# docker info | grep -i cgroup
Cgroup Driver: cgroupfs

Then install the kubelet, this would solve your problem.

Answer 5

To get minicube running on a CentOS 7, I need to start it with a --extra-config=kubelet.cgroup-driver=systemd as suggested in https://github.com/kubernetes/minikube/issues/2192.