delete kubernetes cluster on docker-for-desktop OS

问题:

What is the equivalent command for minikube delete in docker-for-desktop on OSX

As I understand, minikube creates a VM to host its kubernetes cluster but I do not understand how docker-for-desktop is managing this on OSX.

回答1:

Tear down Kubernetes in Docker for OS X is quite an easy task.

Go to Preferences, open Reset tab, and click Reset Kubernetes cluster.

All object that have been created with Kubectl before that will be deleted.

You can also reset docker VM image (Reset disk image) and all settings (Reset to factory defaults) or even uninstall Docker.

回答2:

In recent Docker Edge versions for Mac ( 2.1.7 ) Preferences design has been changed. Now you can reset Kubernetes cluster and other docker aspects by switching to the bug plane in the top right of Preferences window:

Note: You are able to reset Kubernetes cluster only if it's enabled. If you uncheck "Enable Kubernetes" checkbox, "Reset Kubernetes cluster" button becomes inactive.

For convenience "Reset Kubernetes cluster" is also present on the Kubernetes tab in the main Preferences plane:

回答3:

It's really under the hood in the code. Docker for Mac uses these components: Hyperkit, VPNkit and DataKit

Kubernetes runs in the same Hyperkit VM created for docker and the kube-apiserver is exposed.

You can connect to the VM with this:

screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty

Then you can see all the Kubernetes processes in the VM:

linuxkit-025000000001:~# ps -Af | grep kube
 1251 root      0:00 /usr/bin/logwrite -n kubelet /usr/bin/kubelet.sh
 1288 root      0:51 kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --cgroups-per-qos=false --enforce-node-allocatable= --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cadvisor-port=0 --kube-reserved-cgroup=podruntime --system-reserved-cgroup=systemreserved --cgroup-root=kubepods --hostname-override=docker-for-desktop --fail-swap-on=false
 3564 root      0:26 kube-scheduler --address=127.0.0.1 --leader-elect=true --kubeconfig=/etc/kubernetes/scheduler.conf
 3616 root      1:45 kube-controller-manager --cluster-signing-key-file=/run/config/pki/ca.key --address=127.0.0.1 --root-ca-file=/run/config/pki/ca.crt --service-account-private-key-file=/run/config/pki/sa.key --kubeconfig=/etc/kubernetes/controller-manager.conf --cluster-signing-cert-file=/run/config/pki/ca.crt --leader-elect=true --use-service-account-credentials=true --controllers=*,bootstrapsigner,tokencleaner
 3644 root      1:59 kube-apiserver --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --service-account-key-file=/run/config/pki/sa.pub --secure-port=6443 --insecure-port=8080 --insecure-bind-address=0.0.0.0 --requestheader-client-ca-file=/run/config/pki/front-proxy-ca.crt --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-extra-headers-prefix=X-Remote-Extra- --advertise-address=192.168.65.3 --service-cluster-ip-range=10.96.0.0/12 --tls-private-key-file=/run/config/pki/apiserver.key --enable-bootstrap-token-auth=true --requestheader-allowed-names=front-proxy-client --tls-cert-file=/run/config/pki/apiserver.crt --proxy-client-key-file=/run/config/pki/front-proxy-client.key --proxy-client-cert-file=/run/config/pki/front-proxy-client.crt --allow-privileged=true --client-ca-file=/run/config/pki/ca.crt --kubelet-client-certificate=/run/config/pki/apiserver-kubelet-client.crt --kubelet-client-key=/run/config/pki/apiserver-kubelet-client.key --authorization-mode=Node,RBAC --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/run/config/pki/etcd/ca.crt --etcd-certfile=/run/config/pki/apiserver-etcd-client.crt --etcd-keyfile=/run/config/pki/apiserver-etcd-client.key
 3966 root      0:01 /kube-dns --domain=cluster.local. --dns-port=10053 --config-dir=/kube-dns-config --v=2
 4190 root      0:05 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf
 4216 65534     0:03 /sidecar --v=2 --logtostderr --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,SRV --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,SRV
 4606 root      0:00 /compose-controller --kubeconfig  --reconciliation-interval 30s
 4905 root      0:01 /api-server --kubeconfig  --authentication-kubeconfig  --authorization-kubeconfig  --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/etc/docker-compose/etcd/ca.crt --etcd-certfile=/etc/docker-compose/etcd/client.crt --etcd-keyfile=/etc/docker-compose/etcd/client.key --secure-port=9443 --tls-ca-file=/etc/docker-compose/tls/ca.crt --tls-cert-file=/etc/docker-compose/tls/server.crt --tls-private-key-file=/etc/docker-compose/tls/server.key

So if you uncheck the following box (unclear from the docs what command it uses):

You can see that the processes are removed:

linuxkit-025000000001:~# [ 6616.856404] cni0: port 2(veth5f6c8b28) entered disabled state
[ 6616.860520] device veth5f6c8b28 left promiscuous mode
[ 6616.861125] cni0: port 2(veth5f6c8b28) entered disabled state

linuxkit-025000000001:~#
linuxkit-025000000001:~# [ 6626.816763] cni0: port 1(veth87e77142) entered disabled state
[ 6626.822748] device veth87e77142 left promiscuous mode
[ 6626.823329] cni0: port 1(veth87e77142) entered disabled state

linuxkit-025000000001:~# ps -Af | grep kube
linuxkit-025000000001:~#

回答4:

To reset Docker-desktop Kubernetes cluster using command line, put the following content to a file (dd-reset.sh) and mark it executable ( chmod a+x dd-reset.sh )

#!/bin/bash

dr='docker run -it --rm --privileged --pid=host debian nsenter -t 1 -m -u -n -i'

${dr} sh -c 'export PATH=$PATH:/containers/services/docker/rootfs/usr/bin:/containers/services/docker/rootfs/usr/local/bin:/var/lib/kube-binary-cache/ && \
if [ ! -e /var/run/docker.sock ] ; then ln -s /containers/services/docker/rootfs/var/run/docker.sock /var/run/docker.sock ; fi && \
kube-reset.sh'

sleep 3

echo "cluster resetted. restarting docker-desktop..."
osascript -e 'quit app "Docker"'

open --background -a Docker
echo "docker-desktop started. Wait 3-5 mins for kubernetes to start."

Explanation:

This method uses internal scripts from Docker-desktop VM. To make it work, some preparation of user environment is required.

I wasn't able to start Kubernetes cluster using kube-start.sh script from inside the VM, so I've used MacOS commands to restart Docker application instead.

This method works even if your Kubernetes cluster is not enabled in Docker preferences at the moment, but it's required to enable Kubernetes at least once to use the script.

It was tested on Docker Edge for MacOS v2.2.2.0 (43066)
There is no guarantee that it will be compatible with earlier or later versions.

This version of Docker uses kubeadm to initialize Kubernetes cluster. Scripts are located in the folder /containers/services/docker/rootfs/usr/bin:

• kube-pull.sh (brings kubernetes binaries to VM)
• kube-reset.sh (runs kube-stop.sh and do kubeadm reset + some rm stuff)
• kube-restart.sh (runs kube-stop.sh and kube-start.sh)
• kube-start.sh (runs kube-pull.sh and kubelet.sh)
• kube-stop.sh (kills kubelet and kube-apiserver processes, and all k8s containers)
• kubeadm-init.sh (initializes Kubernetes cluster)
• kubelet.sh (runs kubeadm-init.sh and starts kubelet binary)

Cluster configuration is located in the file /containers/services/docker/lower/etc/kubeadm/kubeadm.yaml

Resources used:

• Restart Docker from command line
• Use nsenter in priviledged container

回答5:

Just delete the vm that holds the kubernetes resources.

$ minikube delete