如何诊断Kubernetes没有响应的API(How to diagnose Kubernetes

2019-10-29 15:48发布

站内文章 / 前端开发
164 0 0 
Aug 01 12:49:49 master kubelet[18344]: E0801 12:49:49.534129   18344 eviction_manager.go:243] eviction manager: failed to get get summary stats: failed to get node info: node "master" not found
Aug 01 12:49:49 master kubelet[18344]: I0801 12:49:49.925152   18344 kubelet_node_status.go:269] Setting node annotation to enable volume controller attach/detach
Aug 01 12:49:49 master kubelet[18344]: I0801 12:49:49.927988   18344 kubelet_node_status.go:79] Attempting to register node master
Aug 01 12:49:49 master kubelet[18344]: E0801 12:49:49.928908   18344 kubelet_node_status.go:103] Unable to register node "master" with API server: Post https://192.168.0.33:6443/api/v1/nodes: dial tcp 192.168.
Aug 01 12:49:50 master kubelet[18344]: E0801 12:49:50.004760   18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://192.168.0.33:6443/api/v1/nodes?fieldSel
Aug 01 12:49:50 master kubelet[18344]: E0801 12:49:50.006130   18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://192.168.0.33:6443/api/v1/services?li
Aug 01 12:49:50 master kubelet[18344]: E0801 12:49:50.008020   18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.0.33:6443/api/v1/pods?fi
Aug 01 12:49:50 master kubelet[18344]: I0801 12:49:50.729713   18344 kubelet_node_status.go:269] Setting node annotation to enable volume controller attach/detach
Aug 01 12:49:50 master kubelet[18344]: I0801 12:49:50.733513   18344 kubelet_node_status.go:79] Attempting to register node master
Aug 01 12:49:50 master kubelet[18344]: E0801 12:49:50.734866   18344 kubelet_node_status.go:103] Unable to register node "master" with API server: Post https://192.168.0.33:6443/api/v1/nodes: dial tcp 192.168.
Aug 01 12:49:51 master kubelet[18344]: E0801 12:49:51.006313   18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://192.168.0.33:6443/api/v1/nodes?fieldSel
Aug 01 12:49:51 master kubelet[18344]: E0801 12:49:51.009443   18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://192.168.0.33:6443/api/v1/services?li
Aug 01 12:49:51 master kubelet[18344]: E0801 12:49:51.010510   18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.0.33:6443/api/v1/pods?fi
lines 22914-22948/22948

我怎样才能进一步诊断呢？ 

telnet 192.168.0.33 6443
Trying 192.168.0.33...
telnet: Unable to connect to remote host: Connection refused

systemctl状态kubelet.service 

● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Wed 2018-08-01 12:49:48 EDT; 3min 47s ago
     Docs: http://kubernetes.io/docs/
 Main PID: 18344 (kubelet)
    Tasks: 13 (limit: 4915)
   Memory: 39.4M
      CPU: 4.091s
   CGroup: /system.slice/kubelet.service
           └─18344 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cn

Aug 01 12:53:33 master kubelet[18344]: E0801 12:53:33.522282   18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://192.168.0.33:6443/api/v1/services?limit=500&resourceVersion=0
Aug 01 12:53:33 master kubelet[18344]: E0801 12:53:33.527787   18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://192.168.0.33:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmas
Aug 01 12:53:33 master kubelet[18344]: E0801 12:53:33.537549   18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.0.33:6443/api/v1/pods?fieldSelector=spec.nodeName
Aug 01 12:53:34 master kubelet[18344]: I0801 12:53:34.051830   18344 kubelet_node_status.go:269] Setting node annotation to enable volume controller attach/detach
Aug 01 12:53:34 master kubelet[18344]: E0801 12:53:34.523429   18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://192.168.0.33:6443/api/v1/services?limit=500&resourceVersion=0
Aug 01 12:53:34 master kubelet[18344]: E0801 12:53:34.530208   18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://192.168.0.33:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmas
Aug 01 12:53:34 master kubelet[18344]: E0801 12:53:34.538744   18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.0.33:6443/api/v1/pods?fieldSelector=spec.nodeName
Aug 01 12:53:35 master kubelet[18344]: E0801 12:53:35.524380   18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://192.168.0.33:6443/api/v1/services?limit=500&resourceVersion=0
Aug 01 12:53:35 master kubelet[18344]: E0801 12:53:35.531218   18344 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://192.168.0.33:6443/api/v1/nodes?fieldSelecto

我想我打破的东西...

Answer 1:

耶我发现API服务器日志。 

osboxes@master:/var/log/pods$ sudo docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS               NAMES
84ec4c4de5b2        k8s.gcr.io/pause:3.1   "/pause"                 About an hour ago   Up About an hour                        k8s_POD_kube-apiserver-master_kube-system_fdb932ada5768a1891d839f8cf2306a9_0
ea84510e26be        272b3a60cd68           "kube-scheduler --..."   About an hour ago   Up About an hour                        k8s_kube-scheduler_kube-scheduler-master_kube-system_537879acc30dd5eff5497cb2720a6d64_1
d36af3896c3b        52096ee87d0e           "kube-controller-m..."   About an hour ago   Up About an hour                        k8s_kube-controller-manager_kube-controller-manager-master_kube-system_f31b9af1b177e27c1d4ace1fa4d37d83_1
a53569da6f29        f0fad859c909           "/opt/bin/flanneld..."   4 hours ago         Up 4 hours                              k8s_kube-flannel_kube-flannel-ds-94xm7_kube-system_7e32bed6-9585-11e8-a2f7-080027a08edc_0
9a31712003ac        k8s.gcr.io/pause:3.1   "/pause"                 4 hours ago         Up 4 hours                              k8s_POD_kube-flannel-ds-94xm7_kube-system_7e32bed6-9585-11e8-a2f7-080027a08edc_0
276b6107a4b3        d5c25579d0ff           "/usr/local/bin/ku..."   4 hours ago         Up 4 hours                              k8s_kube-proxy_kube-proxy-2hcfx_kube-system_e7bebd1e-9584-11e8-a2f7-080027a08edc_0
87d2c5657240        k8s.gcr.io/pause:3.1   "/pause"                 4 hours ago         Up 4 hours                              k8s_POD_kube-proxy-2hcfx_kube-system_e7bebd1e-9584-11e8-a2f7-080027a08edc_0
d04c7669f27c        b8df3b177be2           "etcd --advertise-..."   4 hours ago         Up 4 hours                              k8s_etcd_etcd-master_kube-system_d3a295b6d0da8bbfe30c134cab4d030b_0
6c174ea2f877        k8s.gcr.io/pause:3.1   "/pause"                 4 hours ago         Up 4 hours                              k8s_POD_kube-scheduler-master_kube-system_537879acc30dd5eff5497cb2720a6d64_0
e5603c531a1c        k8s.gcr.io/pause:3.1   "/pause"                 4 hours ago         Up 4 hours                              k8s_POD_kube-controller-manager-master_kube-system_f31b9af1b177e27c1d4ace1fa4d37d83_0
cf6ee3a78089        k8s.gcr.io/pause:3.1   "/pause"                 4 hours ago         Up 4 hours                              k8s_POD_etcd-master_kube-system_d3a295b6d0da8bbfe30c134cab4d030b_0
osboxes@master:/var/log/pods$ tail -f fdb932ada5768a1891d839f8cf2306a9/kube-apiserver/29.log 
tail: cannot open 'fdb932ada5768a1891d839f8cf2306a9/kube-apiserver/29.log' for reading: Permission denied
tail: no files remaining
osboxes@master:/var/log/pods$ tail -f fdb932ada5768a1891d839f8cf2306a9/kube-apiserver/29.log 
tail: cannot open 'fdb932ada5768a1891d839f8cf2306a9/kube-apiserver/29.log' for reading: Permission denied
tail: no files remaining
osboxes@master:/var/log/pods$ sudo tail -f fdb932ada5768a1891d839f8cf2306a9/kube-apiserver/29.log 
{"log":"      --tls-private-key-file string                             File containing the default x509 private key matching --tls-cert-file.\n","stream":"stderr","time":"2018-08-01T16:56:17.315279517Z"}
{"log":"      --tls-sni-cert-key namedCertKey                           A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: \"example.crt,example.key\" or \"foo.crt,foo.key:*.foo.com,foo.com\". (default [])\n","stream":"stderr","time":"2018-08-01T16:56:17.315282242Z"}
{"log":"      --token-auth-file string                                  If set, the file that will be used to secure the secure port of the API server via token authentication.\n","stream":"stderr","time":"2018-08-01T16:56:17.31528618Z"}
{"log":"  -v, --v Level                                                 log level for V logs\n","stream":"stderr","time":"2018-08-01T16:56:17.315288985Z"}
{"log":"      --version version[=true]                                  Print version information and quit\n","stream":"stderr","time":"2018-08-01T16:56:17.31529159Z"}
{"log":"      --vmodule moduleSpec                                      comma-separated list of pattern=N settings for file-filtered logging\n","stream":"stderr","time":"2018-08-01T16:56:17.315296449Z"}
{"log":"      --watch-cache                                             Enable watch caching in the apiserver (default true)\n","stream":"stderr","time":"2018-08-01T16:56:17.315299265Z"}
{"log":"      --watch-cache-sizes strings                               List of watch cache sizes for every resource (pods, nodes, etc.), comma separated. The individual override format: resource[.group]#size, where resource is lowercase plural (no version), group is optional, and size is a number. It takes effect when watch-cache is enabled. Some resources (replicationcontrollers, endpoints, nodes, pods, services, apiservices.apiregistration.k8s.io) have system defaults set by heuristics, others default to default-watch-cache-size\n","stream":"stderr","time":"2018-08-01T16:56:17.31530237Z"}
{"log":"\n","stream":"stderr","time":"2018-08-01T16:56:17.315306007Z"}
{"log":"error: loading audit policy file: failed to read file path \"/etc/kubernetes/audit.yaml\": open /etc/kubernetes/audit.yaml: no such file or directory\n","stream":"stderr","time":"2018-08-01T16:56:17.315480703Z"}
q^C


文章来源: How to diagnose Kubernetes not responding on API
标签： kubernetes
举报

收藏的人(0)

Ta的文章 更多文章
登录 后发表评论
0条评论
还没有人评论过~