kubernetes: cannot access NodePort from other mach

2020-02-12 07:09发布

站内问答 / Kubernetes
660 4 4

I am trying to install kubernetes. There was a problem when checking the container executed on kubernetes. I set the type of service to Nodeport, but I could not access from a node other than the one on which the container is running. Want to make it accessible from other computers, please tell me where it is different. I tried externalIPs and LoadBarancer, but it was impossible.

Enviroment

  • OS:Ubuntu 16.04 LTS
  • Kubernetes:1.8
  • Docker:17.09.0-ce
  • etcd:3.2.8
  • flannel:0.9.0

Network

  • Physical:10.1.1.0/24
  • flannel:172.16.0.0/16
  • docker:192.168.0.0/16

Machines

  • Master Node(2nodes):10.1.1.24,10.1.1.25
  • Worker Node(2nodes):10.1.1.26,10.1.1.27

kubectl describe svc nginx-cluster

Name:              nginx-cluster
Namespace:         default
Labels:            app=nginx-demo
Annotations:       <none>
Selector:          app=nginx-demo
Type:              ClusterIP
IP:                172.16.236.159
Port:              <unset>  8090/TCP
TargetPort:        80/TCP
Endpoints:         192.168.24.2:80
Session Affinity:  None
Events:            <none>

kubectl describe svc nginx-service

Name:                     nginx-service
Namespace:                default
Labels:                   app=nginx-demo
Annotations:              <none>
Selector:                 app=nginx-demo
Type:                     NodePort
IP:                       172.16.199.69
Port:                     <unset>  8090/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  31659/TCP
Endpoints:                192.168.24.2:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

running container worker node(10.1.1.27)

curl 10.1.1.27:31659

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

worker node(10.1.1.26)

curl 10.1.1.27:31659

curl: (7) Failed to connect to 10.1.1.27 port 31659:Connection timed out.

other machine(10.1.1.XX)

curl 10.1.1.27:31659

curl: (7) Failed to connect to 10.1.1.27 port 31659:Connection timed out.

kubectl get pods -o wide

NAME                          READY     STATUS    RESTARTS   AGE       IP             NODE
echoserver-848b75d85-9fx7r    1/1       Running   3          6d        192.168.70.2   k8swrksv01
nginx-demo-85cc49574c-wv2b9   1/1       Running   3          6d        192.168.2.2    k8swrksv02

kubectl get svc -o wide

NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE       SELECTOR
clusterip       ClusterIP   172.16.39.77     <none>        80/TCP           6d        run=echoserver
kubernetes      ClusterIP   172.16.0.1       <none>        443/TCP          10d       <none>
nginx-cluster   ClusterIP   172.16.236.159   <none>        8090/TCP         6d        app=nginx-demo
nginx-service   NodePort    172.16.199.69    <none>        8090:31659/TCP   6d        app=nginx-demo
nodeport        NodePort    172.16.38.40     <none>        80:31317/TCP     6d        run=echoserver

netstat -ntlp

tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      1963/kubelet
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      2202/kube-proxy
tcp        0      0 127.0.0.1:4243          0.0.0.0:*               LISTEN      1758/dockerd
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      996/sshd
tcp6       0      0 :::4194                 :::*                    LISTEN      1963/kubelet
tcp6       0      0 :::10250                :::*                    LISTEN      1963/kubelet
tcp6       0      0 :::31659                :::*                    LISTEN      2202/kube-proxy
tcp6       0      0 :::10255                :::*                    LISTEN      1963/kubelet
tcp6       0      0 :::10256                :::*                    LISTEN      2202/kube-proxy
tcp6       0      0 :::31317                :::*                    LISTEN      2202/kube-proxy
tcp6       0      0 :::22                   :::*                    LISTEN      996/sshd

iptables-save

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-AZ4EGFEAU4RTSLJO - [0:0]
:KUBE-SEP-C7HQKKO26GIFOZZM - [0:0]
:KUBE-SEP-EWKNS2YCPXGJCXDC - [0:0]
:KUBE-SEP-LQVPUPFGW6BWATIP - [0:0]
:KUBE-SEP-OMMOFZ27GPKZ4OPA - [0:0]
:KUBE-SEP-UD3HOGDD5NDLNY74 - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-CQNAS6RSUGJF2C2D - [0:0]
:KUBE-SVC-GKN7Y2BSGW4NJTYL - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-XP7QDA4CRQ2QA33W - [0:0]
:KUBE-SVC-Z5P6OMNAEVLAQUTS - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 192.168.2.0/24 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 192.168.0.0/16 -d 192.168.0.0/16 -j RETURN
-A POSTROUTING -s 192.168.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
-A POSTROUTING ! -s 192.168.0.0/16 -d 192.168.2.0/24 -j RETURN
-A POSTROUTING ! -s 192.168.0.0/16 -d 192.168.0.0/16 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx-service:" -m tcp --dport 31659 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx-service:" -m tcp --dport 31659 -j KUBE-SVC-GKN7Y2BSGW4NJTYL
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport:" -m tcp --dport 31317 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport:" -m tcp --dport 31317 -j KUBE-SVC-XP7QDA4CRQ2QA33W
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-AZ4EGFEAU4RTSLJO -s 192.168.70.2/32 -m comment --comment "default/clusterip:" -j KUBE-MARK-MASQ
-A KUBE-SEP-AZ4EGFEAU4RTSLJO -p tcp -m comment --comment "default/clusterip:" -m tcp -j DNAT --to-destination 192.168.70.2:8080
-A KUBE-SEP-C7HQKKO26GIFOZZM -s 192.168.70.2/32 -m comment --comment "default/nodeport:" -j KUBE-MARK-MASQ
-A KUBE-SEP-C7HQKKO26GIFOZZM -p tcp -m comment --comment "default/nodeport:" -m tcp -j DNAT --to-destination 192.168.70.2:8080
-A KUBE-SEP-EWKNS2YCPXGJCXDC -s 10.1.1.25/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-EWKNS2YCPXGJCXDC -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-EWKNS2YCPXGJCXDC --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.1.1.25:6443
-A KUBE-SEP-LQVPUPFGW6BWATIP -s 192.168.2.2/32 -m comment --comment "default/nginx-service:" -j KUBE-MARK-MASQ
-A KUBE-SEP-LQVPUPFGW6BWATIP -p tcp -m comment --comment "default/nginx-service:" -m tcp -j DNAT --to-destination 192.168.2.2:80
-A KUBE-SEP-OMMOFZ27GPKZ4OPA -s 10.1.1.24/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-OMMOFZ27GPKZ4OPA -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-OMMOFZ27GPKZ4OPA --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.1.1.24:6443
-A KUBE-SEP-UD3HOGDD5NDLNY74 -s 192.168.2.2/32 -m comment --comment "default/nginx-cluster:" -j KUBE-MARK-MASQ
-A KUBE-SEP-UD3HOGDD5NDLNY74 -p tcp -m comment --comment "default/nginx-cluster:" -m tcp -j DNAT --to-destination 192.168.2.2:80
-A KUBE-SERVICES -d 172.16.236.159/32 -p tcp -m comment --comment "default/nginx-cluster: cluster IP" -m tcp --dport 8090 -j KUBE-SVC-Z5P6OMNAEVLAQUTS
-A KUBE-SERVICES -d 172.16.199.69/32 -p tcp -m comment --comment "default/nginx-service: cluster IP" -m tcp --dport 8090 -j KUBE-SVC-GKN7Y2BSGW4NJTYL
-A KUBE-SERVICES -d 172.16.38.40/32 -p tcp -m comment --comment "default/nodeport: cluster IP" -m tcp --dport 80 -j KUBE-SVC-XP7QDA4CRQ2QA33W
-A KUBE-SERVICES -d 172.16.39.77/32 -p tcp -m comment --comment "default/clusterip: cluster IP" -m tcp --dport 80 -j KUBE-SVC-CQNAS6RSUGJF2C2D
-A KUBE-SERVICES -d 172.16.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-CQNAS6RSUGJF2C2D -m comment --comment "default/clusterip:" -j KUBE-SEP-AZ4EGFEAU4RTSLJO
-A KUBE-SVC-GKN7Y2BSGW4NJTYL -m comment --comment "default/nginx-service:" -j KUBE-SEP-LQVPUPFGW6BWATIP
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-OMMOFZ27GPKZ4OPA --mask 255.255.255.255 --rsource -j KUBE-SEP-OMMOFZ27GPKZ4OPA
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-EWKNS2YCPXGJCXDC --mask 255.255.255.255 --rsource -j KUBE-SEP-EWKNS2YCPXGJCXDC
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-OMMOFZ27GPKZ4OPA
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-EWKNS2YCPXGJCXDC
-A KUBE-SVC-XP7QDA4CRQ2QA33W -m comment --comment "default/nodeport:" -j KUBE-SEP-C7HQKKO26GIFOZZM
-A KUBE-SVC-Z5P6OMNAEVLAQUTS -m comment --comment "default/nginx-cluster:" -j KUBE-SEP-UD3HOGDD5NDLNY74
COMMIT
*filter
:INPUT ACCEPT [40:14606]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [42:6275]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:DOCKER-USER - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION -j RETURN
-A DOCKER-USER -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
COMMIT

标签： kubernetes
举报
4条回答
再贱就再见
2楼-- · 2020-02-12 07:38

additional information. Something is supposed to block the port, but that is unknown...

pod running node

nmap 10.1.1.27 -p31000-32000

Not shown: 999 closed ports
PORT      STATE SERVICE
31317/tcp open  unknown
31659/tcp open  unknown

other node

nmap 10.1.1.27 -p31000-32000

Not shown: 999 closed ports
PORT      STATE    SERVICE
31317/tcp filtered unknown
31659/tcp filtered unknown
查看更多
0人赞 添加讨论(0) 举报
Deceive 欺骗
3楼-- · 2020-02-12 07:42

It's caused by default DROP on FORWARD chain (which in turn is caused by docker).

You can see it working again if you add iptables -A FORWARD -j ACCEPT rule to your node.

The k8s issue is here: https://github.com/kubernetes/kubernetes/issues/39823 but the actual fix is here https://github.com/kubernetes/kubernetes/pull/52569 (expected to be in 1.9).

查看更多
0人赞 添加讨论(0) 举报
劫难
4楼-- · 2020-02-12 08:02

Farcaller is right. We hit the same on centos running firewalld.

Until we upgrade to k8s 1.9 we added the following firewalld rule. The rule is similar to the one created by kube-proxy in k8s 1.9

#!/bin/bash
# follows https://github.com/kubernetes/kubernetes/pull/52569 introduced in k8s 1.9
# required to support nodeport services routing from all nodes in the cluster when the firewall is turned on.
# KUBE-MARK-MASQ corresponds to kube-proxy --iptables-masquerade-bit=14, which is the default.
KUBE_MARK_MASQ="0x4000/0x4000"
firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 3 -m mark --mark "$KUBE_MARK_MASQ" -j ACCEPT
查看更多
0人赞 添加讨论(0) 举报
倾城　Initia
5楼-- · 2020-02-12 08:04

you can first check the node port whether the port is open,

use

netstat -ntlp

to check, if yes , that maybe some problem with the iptable or routing,

if no , please check the firewall or other question

Good luck

查看更多
0人赞 添加讨论(0) 举报
登录 后发表回答
相关问题
查看全部
相关文章
查看全部
收藏的人(4)