route not working in kubernetes with calico

Question

I have

• kubernetes v1.6.0 setup by kubeadm v1.6.1
• calico setup by offical yaml
• iptables v1.6.0
• nodes are provided by AliCloud

Problem:

The cni network is not working. Any deployment can only be visited from the node where it is running. I doubt it is related with route table conflict/missing, because I have another cluster on Vultr Cloud working fine, with the same setup steps.

Cluster Info:

root@iZ2ze8ctk2q17u029a8wcoZ:~# kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                              READY     STATUS    RESTARTS   AGE       IP               NODE
kube-system   calico-etcd-66gf4                                 1/1       Running   0          16h       10.27.219.50     iz2ze8ctk2q17u029a8wcoz
kube-system   calico-node-4wxsb                                 2/2       Running   0          16h       10.27.219.50     iz2ze8ctk2q17u029a8wcoz
kube-system   calico-node-6n1g1                                 2/2       Running   0          16h       10.30.248.80     iz2zegw6nmd5t5qxy35lh0z
kube-system   calico-policy-controller-2561685917-7bdd4         1/1       Running   0          16h       10.30.248.80     iz2zegw6nmd5t5qxy35lh0z
kube-system   etcd-iz2ze8ctk2q17u029a8wcoz                      1/1       Running   0          16h       10.27.219.50     iz2ze8ctk2q17u029a8wcoz
kube-system   heapster-bx03l                                    1/1       Running   0          16h       192.168.31.150   iz2zegw6nmd5t5qxy35lh0z
kube-system   kube-apiserver-iz2ze8ctk2q17u029a8wcoz            1/1       Running   0          16h       10.27.219.50     iz2ze8ctk2q17u029a8wcoz
kube-system   kube-controller-manager-iz2ze8ctk2q17u029a8wcoz   1/1       Running   0          16h       10.27.219.50     iz2ze8ctk2q17u029a8wcoz
kube-system   kube-dns-3913472980-kgzln                         3/3       Running   0          16h       192.168.31.149   iz2zegw6nmd5t5qxy35lh0z
kube-system   kube-proxy-ck83t                                  1/1       Running   0          16h       10.30.248.80     iz2zegw6nmd5t5qxy35lh0z
kube-system   kube-proxy-lssdn                                  1/1       Running   0          16h       10.27.219.50     iz2ze8ctk2q17u029a8wcoz
kube-system   kube-scheduler-iz2ze8ctk2q17u029a8wcoz            1/1       Running   0          16h       10.27.219.50     iz2ze8ctk2q17u029a8wcoz

I checked each pod's log, cannot find anything wrong.

Master Info: internal ip: 10.27.219.50

root@iZ2ze8ctk2q17u029a8wcoZ:~# ifconfig

docker0   Link encap:Ethernet  HWaddr 02:42:56:84:35:19
          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth0      Link encap:Ethernet  HWaddr 00:16:3e:30:51:ae
          inet addr:10.27.219.50  Bcast:10.27.219.255  Mask:255.255.252.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4400927 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3906530 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:564808928 (564.8 MB)  TX bytes:792611382 (792.6 MB)

eth1      Link encap:Ethernet  HWaddr 00:16:3e:32:07:f8
          inet addr:59.110.32.199  Bcast:59.110.35.255  Mask:255.255.252.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1148756 errors:0 dropped:0 overruns:0 frame:0
          TX packets:688177 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1570341044 (1.5 GB)  TX bytes:58104611 (58.1 MB)

tunl0     Link encap:IPIP Tunnel  HWaddr
          inet addr:192.168.201.0  Mask:255.255.255.255
          UP RUNNING NOARP  MTU:1440  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)


root@iZ2ze8ctk2q17u029a8wcoZ:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         59.110.35.247   0.0.0.0         UG    0      0        0 eth1
10.27.216.0     0.0.0.0         255.255.252.0   U     0      0        0 eth0
10.30.0.0       10.27.219.247   255.255.0.0     UG    0      0        0 eth0
10.32.0.0       0.0.0.0         255.240.0.0     U     0      0        0 weave
59.110.32.0     0.0.0.0         255.255.252.0   U     0      0        0 eth1
100.64.0.0      10.27.219.247   255.192.0.0     UG    0      0        0 eth0
172.16.0.0      10.27.219.247   255.240.0.0     UG    0      0        0 eth0
172.17.0.0      0.0.0.0         255.255.255.0   U     0      0        0 docker0
192.168.201.0   0.0.0.0         255.255.255.192 U     0      0        0 *

root@iZ2ze8ctk2q17u029a8wcoZ:~# ip route list
default via 59.110.35.247 dev eth1
10.27.216.0/22 dev eth0  proto kernel  scope link  src 10.27.219.50
10.30.0.0/16 via 10.27.219.247 dev eth0
10.32.0.0/12 dev weave  proto kernel  scope link  src 10.32.0.1
59.110.32.0/22 dev eth1  proto kernel  scope link  src 59.110.32.199
100.64.0.0/10 via 10.27.219.247 dev eth0
172.16.0.0/12 via 10.27.219.247 dev eth0
172.17.0.0/24 dev docker0  proto kernel  scope link  src 172.17.0.1 linkdown
blackhole 192.168.201.0/26  proto bird

// NOTE: 10.30.0.0/16 via 10.27.219.247 dev eth0
// this rule is important, the worker node's ip is 10.30.xx.xx. If I delete this rule, I cannot ping worker node.
// this rule is 10.0.0.0/8 via 10.27.219.247 dev eth0 by default, I changed it to the above.


root@iZ2ze8ctk2q17u029a8wcoZ:~# iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 3 packets, 180 bytes)
 pkts bytes target     prot opt in     out     source               destination
20976 1250K cali-PREROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:6gwbT8clXdHdC1b1 */
21016 1252K KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
20034 1193K DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 3 packets, 180 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 4 packets, 240 bytes)
 pkts bytes target     prot opt in     out     source               destination
 109K 6580K cali-OUTPUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:tVnHkvAo15HuiPy0 */
 111K 6738K KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
 1263 75780 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 4 packets, 240 bytes)
 pkts bytes target     prot opt in     out     source               destination
86584 5235K cali-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:O3lYWMrLQYEMJtB5 */
    0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/24        0.0.0.0/0
3982K  239M KUBE-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
28130 1704K WEAVE      all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0

Chain KUBE-MARK-DROP (0 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK or 0x8000

Chain KUBE-MARK-MASQ (5 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK or 0x4000

Chain KUBE-NODEPORTS (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain KUBE-POSTROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000

Chain KUBE-SEP-2VS52M6CEWASZVOP (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.31.149       0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.31.149:53

Chain KUBE-SEP-3XQHSFTDAPNNNDX3 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.31.150       0.0.0.0/0            /* kube-system/heapster: */
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/heapster: */ tcp to:192.168.31.150:8082

Chain KUBE-SEP-CH7KJM5XKO5WGA6D (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-MARK-MASQ  all  --  *      *       10.27.219.50         0.0.0.0/0            /* default/kubernetes:https */
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */ recent: SET name: KUBE-SEP-CH7KJM5XKO5WGA6D side: source mask: 255.255.255.255 tcp to:10.27.219.50:6443

Chain KUBE-SEP-X3WTOMIYJNS7APAN (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.31.149       0.0.0.0/0            /* kube-system/kube-dns:dns */
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ udp to:192.168.31.149:53

Chain KUBE-SEP-YDCHDMTZNPMRRKCX (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-MARK-MASQ  all  --  *      *       10.27.219.50         0.0.0.0/0            /* kube-system/calico-etcd: */
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/calico-etcd: */ tcp to:10.27.219.50:6666

Chain KUBE-SERVICES (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  *      *       0.0.0.0/0            10.96.0.1            /* default/kubernetes:https cluster IP */ tcp dpt:443
    0     0 KUBE-SVC-TCOU7JCQXEZGVUNU  udp  --  *      *       0.0.0.0/0            10.96.0.10           /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
    0     0 KUBE-SVC-ERIFXISQEP7F7OF4  tcp  --  *      *       0.0.0.0/0            10.96.0.10           /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
    0     0 KUBE-SVC-NTYB37XIWATNM25Y  tcp  --  *      *       0.0.0.0/0            10.96.232.136        /* kube-system/calico-etcd: cluster IP */ tcp dpt:6666
    0     0 KUBE-SVC-BJM46V3U5RZHCFRZ  tcp  --  *      *       0.0.0.0/0            10.96.181.180        /* kube-system/heapster: cluster IP */ tcp dpt:80
    7   420 KUBE-NODEPORTS  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

Chain KUBE-SVC-BJM46V3U5RZHCFRZ (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-SEP-3XQHSFTDAPNNNDX3  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/heapster: */

Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-SEP-2VS52M6CEWASZVOP  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */

Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-SEP-CH7KJM5XKO5WGA6D  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-CH7KJM5XKO5WGA6D side: source mask: 255.255.255.255
    0     0 KUBE-SEP-CH7KJM5XKO5WGA6D  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */

Chain KUBE-SVC-NTYB37XIWATNM25Y (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-SEP-YDCHDMTZNPMRRKCX  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/calico-etcd: */

Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-SEP-X3WTOMIYJNS7APAN  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */

Chain WEAVE (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  *      *       10.32.0.0/12         224.0.0.0/4
    1    93 MASQUERADE  all  --  *      *      !10.32.0.0/12         10.32.0.0/12
    0     0 MASQUERADE  all  --  *      *       10.32.0.0/12        !10.32.0.0/12

Chain cali-OUTPUT (1 references)
 pkts bytes target     prot opt in     out     source               destination
 109K 6580K cali-fip-dnat  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:GBTAv2p5CwevEyJm */

Chain cali-POSTROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination
 109K 6571K cali-fip-snat  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:Z-c7XtVd2Bq7s_hA */
 109K 6571K cali-nat-outgoing  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:nYKhEzDlr11Jccal */
    0     0 MASQUERADE  all  --  *      tunl0   0.0.0.0/0            0.0.0.0/0            /* cali:JHlpT-eSqR1TvyYm */ ADDRTYPE match src-type !LOCAL limit-out ADDRTYPE match src-type LOCAL

Chain cali-PREROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination
20976 1250K cali-fip-dnat  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:r6XmIziWUJsdOK6Z */

Chain cali-fip-dnat (2 references)
 pkts bytes target     prot opt in     out     source               destination

Chain cali-fip-snat (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain cali-nat-outgoing (1 references)
 pkts bytes target     prot opt in     out     source               destination
    4   376 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:Wd76s91357Uv7N3v */ match-set cali4-masq-ipam-pools src ! match-set cali4-all-ipam-pools dst

Worker Node Info: internal ip: 10.30.248.80

ifconfig

docker0   Link encap:Ethernet  HWaddr 02:42:58:2b:b5:39
          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth0      Link encap:Ethernet  HWaddr 00:16:3e:2e:3d:fd
          inet addr:10.30.248.80  Bcast:10.30.251.255  Mask:255.255.252.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3856596 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4253613 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:827402268 (827.4 MB)  TX bytes:510838231 (510.8 MB)

eth1      Link encap:Ethernet  HWaddr 00:16:3e:2c:db:d1
          inet addr:47.93.161.177  Bcast:47.93.163.255  Mask:255.255.252.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:890451 errors:0 dropped:0 overruns:0 frame:0
          TX packets:825607 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1695352720 (1.6 GB)  TX bytes:62341312 (62.3 MB)

tunl0     Link encap:IPIP Tunnel  HWaddr
          inet addr:192.168.31.128  Mask:255.255.255.255
          UP RUNNING NOARP  MTU:1440  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)


root@iZ2zegw6nmd5t5qxy35lh0Z:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         47.93.163.247   0.0.0.0         UG    0      0        0 eth1
10.0.0.0        10.30.251.247   255.0.0.0       UG    0      0        0 eth0
10.30.248.0     0.0.0.0         255.255.252.0   U     0      0        0 eth0
47.93.160.0     0.0.0.0         255.255.252.0   U     0      0        0 eth1
100.64.0.0      10.30.251.247   255.192.0.0     UG    0      0        0 eth0
172.16.0.0      10.30.251.247   255.240.0.0     UG    0      0        0 eth0
172.17.0.0      0.0.0.0         255.255.255.0   U     0      0        0 docker0
192.168.31.128  0.0.0.0         255.255.255.192 U     0      0        0 *
192.168.31.149  0.0.0.0         255.255.255.255 UH    0      0        0 cali3567b3362cc
192.168.31.150  0.0.0.0         255.255.255.255 UH    0      0        0 cali9d04015b0e7

root@iZ2zegw6nmd5t5qxy35lh0Z:~# ip route list
default via 47.93.163.247 dev eth1
10.0.0.0/8 via 10.30.251.247 dev eth0
10.30.248.0/22 dev eth0  proto kernel  scope link  src 10.30.248.80
47.93.160.0/22 dev eth1  proto kernel  scope link  src 47.93.161.177
100.64.0.0/10 via 10.30.251.247 dev eth0
172.16.0.0/12 via 10.30.251.247 dev eth0
172.17.0.0/24 dev docker0  proto kernel  scope link  src 172.17.0.1 linkdown
blackhole 192.168.31.128/26  proto bird
192.168.31.149 dev cali3567b3362cc  scope link
192.168.31.150 dev cali9d04015b0e7  scope link

// NOTE: 10.0.0.0/8 via 10.30.251.247 dev eth0
// I didn't change this one. So it is default now.


root@iZ2zegw6nmd5t5qxy35lh0Z:~# iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
 3524  263K cali-PREROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:6gwbT8clXdHdC1b1 */
 3527  263K KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
 1031 53882 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 4 packets, 240 bytes)
 pkts bytes target     prot opt in     out     source               destination
84174 5099K cali-OUTPUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:tVnHkvAo15HuiPy0 */
85201 5163K KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
    0     0 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 7 packets, 420 bytes)
 pkts bytes target     prot opt in     out     source               destination
76279 4644K cali-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:O3lYWMrLQYEMJtB5 */
    0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/24        0.0.0.0/0
87179 5342K KUBE-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
43815 2646K WEAVE      all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0

Chain KUBE-MARK-DROP (0 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK or 0x8000

Chain KUBE-MARK-MASQ (5 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK or 0x4000

Chain KUBE-NODEPORTS (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain KUBE-POSTROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000

Chain KUBE-SEP-2VS52M6CEWASZVOP (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.31.149       0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.31.149:53

Chain KUBE-SEP-3XQHSFTDAPNNNDX3 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.31.150       0.0.0.0/0            /* kube-system/heapster: */
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/heapster: */ tcp to:192.168.31.150:8082

Chain KUBE-SEP-CH7KJM5XKO5WGA6D (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-MARK-MASQ  all  --  *      *       10.27.219.50         0.0.0.0/0            /* default/kubernetes:https */
    3   180 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */ recent: SET name: KUBE-SEP-CH7KJM5XKO5WGA6D side: source mask: 255.255.255.255 tcp to:10.27.219.50:6443

Chain KUBE-SEP-X3WTOMIYJNS7APAN (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.31.149       0.0.0.0/0            /* kube-system/kube-dns:dns */
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ udp to:192.168.31.149:53

Chain KUBE-SEP-YDCHDMTZNPMRRKCX (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-MARK-MASQ  all  --  *      *       10.27.219.50         0.0.0.0/0            /* kube-system/calico-etcd: */
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/calico-etcd: */ tcp to:10.27.219.50:6666

Chain KUBE-SERVICES (2 references)
 pkts bytes target     prot opt in     out     source               destination
    3   180 KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  *      *       0.0.0.0/0            10.96.0.1            /* default/kubernetes:https cluster IP */ tcp dpt:443
    0     0 KUBE-SVC-TCOU7JCQXEZGVUNU  udp  --  *      *       0.0.0.0/0            10.96.0.10           /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
    0     0 KUBE-SVC-ERIFXISQEP7F7OF4  tcp  --  *      *       0.0.0.0/0            10.96.0.10           /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
    0     0 KUBE-SVC-NTYB37XIWATNM25Y  tcp  --  *      *       0.0.0.0/0            10.96.232.136        /* kube-system/calico-etcd: cluster IP */ tcp dpt:6666
    0     0 KUBE-SVC-BJM46V3U5RZHCFRZ  tcp  --  *      *       0.0.0.0/0            10.96.181.180        /* kube-system/heapster: cluster IP */ tcp dpt:80
    0     0 KUBE-NODEPORTS  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

Chain KUBE-SVC-BJM46V3U5RZHCFRZ (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-SEP-3XQHSFTDAPNNNDX3  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/heapster: */

Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-SEP-2VS52M6CEWASZVOP  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */

Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
 pkts bytes target     prot opt in     out     source               destination
    3   180 KUBE-SEP-CH7KJM5XKO5WGA6D  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-CH7KJM5XKO5WGA6D side: source mask: 255.255.255.255
    0     0 KUBE-SEP-CH7KJM5XKO5WGA6D  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */

Chain KUBE-SVC-NTYB37XIWATNM25Y (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-SEP-YDCHDMTZNPMRRKCX  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/calico-etcd: */

Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-SEP-X3WTOMIYJNS7APAN  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */

Chain WEAVE (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain cali-OUTPUT (1 references)
 pkts bytes target     prot opt in     out     source               destination
84174 5099K cali-fip-dnat  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:GBTAv2p5CwevEyJm */

Chain cali-POSTROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination
86501 5298K cali-fip-snat  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:Z-c7XtVd2Bq7s_hA */
86501 5298K cali-nat-outgoing  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:nYKhEzDlr11Jccal */
    0     0 MASQUERADE  all  --  *      tunl0   0.0.0.0/0            0.0.0.0/0            /* cali:JHlpT-eSqR1TvyYm */ ADDRTYPE match src-type !LOCAL limit-out ADDRTYPE match src-type LOCAL

Chain cali-PREROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination
 3524  263K cali-fip-dnat  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:r6XmIziWUJsdOK6Z */

Chain cali-fip-dnat (2 references)
 pkts bytes target     prot opt in     out     source               destination

Chain cali-fip-snat (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain cali-nat-outgoing (1 references)
 pkts bytes target     prot opt in     out     source               destination
   29  1726 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:Wd76s91357Uv7N3v */ match-set cali4-masq-ipam-pools src ! match-set cali4-all-ipam-pools dst

Answer 1

I'm not sure what the problem is but here are a couple things to consider:

• I am not familiar with AliCloud but sometimes there are special consideration for some cloud providers. For example with GCE the IP-in-IP must be explicitly allowed, http://docs.projectcalico.org/v2.1/getting-started/kubernetes/installation/gce.
• I see the weave interface on your master so I'm wondering if weave could have left something around that is causing a problem.
• Also as was suggested in your issue https://github.com/projectcalico/cni-plugin/issues/314 you should check calicoctl node status on the nodes to see if BGP is working as expected.

Answer 2

Problem is found by calicoctl node status. The calico/node use a public ip to communicate with each other. But nodes in AliCloud are behind a firewall. So they cannot do that via public ip address.

As gunjan5 suggested, I used this env var IP_AUTODETECTION_METHOD to specify the internal interface. Problem solved.