Open port for internal_address in Rancher/Kubernat

Question

I have the following configuration to setup the cluster using Rancher (RKE).

rancher-config.yml

nodes:
  - address: 192.168.88.204
    internal_address: 172.16.22.12
    user: dockeruser
    role: [controlplane,worker,etcd]
  - address: 192.168.88.203
    internal_address: 172.16.32.37
    user: dockeruser
    role: [controlplane,worker,etcd]
  - address: 192.168.88.202
    internal_address: 172.16.42.73
    user: dockeruser
    role: [controlplane,worker,etcd]

services:
  etcd:
    snapshot: true
    creation: 6h
    retention: 24h 

According Rancher Networking, I already open the following port for all nodes(192.168.88.204, 192.168.88.203, 192.168.88.202) as firewall-services.

node-firewall.xml

<?xml version="1.0" encoding="utf-8"?>
<service>
    <port port="2376" protocol="tcp"/>
    <port port="2379" protocol="tcp"/>
    <port port="2380" protocol="tcp"/>
    <port port="8472" protocol="udp"/>
    <port port="9099" protocol="tcp"/>
    <port port="10250" protocol="tcp"/>
    <port port="443" protocol="tcp"/>
    <port port="6443" protocol="tcp"/>
    <port port="8472" protocol="udp"/>
    <port port="6443" protocol="tcp"/>
    <port port="10254" protocol="tcp"/>
    <port port="30000-32767" protocol="tcp"/>
</service>

-> commmend

firewall-offline-cmd --new-service-from-file=node-firewall.xml --name=node-firewall
firewall-cmd --reload
firewall-cmd --add-service node-firewall

My RKE is installed on 192.168.88.151. For RKE ->

rancher-firewall.xml

<?xml version="1.0" encoding="utf-8"?>
<service>
    <port port="80" protocol="tcp"/>
    <port port="433" protocol="tcp"/>
    <port port="22" protocol="tcp"/>
    <port port="2376" protocol="tcp"/>
    <port port="6443" protocol="tcp"/>
</service>
firewall-offline-cmd --new-service-from-file=rancher-firewall.xml --name=rancher-firewall
firewall-cmd --reload
firewall-cmd --add-service rancher-firewall

So, I run the following commend to up my RKE

rke up --config ./rancher-config.yml

log is

[root@localhost ~]# rke up --config ./rancher-config.yml
INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [192.168.88.204]
INFO[0000] [dialer] Setup tunnel for host [192.168.88.203]
INFO[0000] [dialer] Setup tunnel for host [192.168.88.202]
INFO[0001] [network] Deploying port listener containers
INFO[0001] [network] Port listener containers deployed successfully
INFO[0001] [network] Running etcd <-> etcd port checks
INFO[0001] [network] Successfully started [rke-port-checker] container on host [192.168.88.202]
INFO[0001] [network] Successfully started [rke-port-checker] container on host [192.168.88.204]
INFO[0001] [network] Successfully started [rke-port-checker] container on host [192.168.88.203]
FATA[0016] [network] Host [192.168.88.202] is not able to connect to the following ports: 
            [172.16.22.12:2379, 172.16.22.12:2380, 172.16.32.37:2379, 172.16.32.37:2380, 172.16.42.73:2380, 172.16.42.73:2379]. 
            Please check network policies and firewall rules

My question is how to open the port for the internal_address for all nodes in kubernates cluster?

Answer 1

May be it is lack of my experience. I just share what I found. internal_address is have to be ip-address of (Gateway) of docker. To know the ip-address of docker for each node (192.168.88.204, 192.168.88.203, 192.168.88.202).

Run the commend docker network ls. You might be get following network information.

NETWORK ID          NAME                DRIVER              SCOPE
aa13d08f2676        bridge              bridge              local
02eabe818790        host                host                local
1e5bb430d790        none                null                local

And run the commend docker network inspect bridge to get ip-addres of bridge. you will get the following similer info.

[
    {
        "Name": "bridge",
        "Id": "aa13d08f2676e40df5a82521fccc4e402ef6b04f82bcd414cd065a1859b3799d",
        "Created": "2019-01-31T21:32:02.381082005-05:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        ....
        ...
        ..
        .

]

and configure rancher-config.yml as below and run rke up --config ./rancher-config.yml again

nodes:
  - address: 192.168.88.204
    internal_address: 172.17.0.1
    ...
...
..
..

Answer 2

That is incorrect.

The internal_address provides the ability to have nodes with multiple addresses set a specific address to use for inter-host communication on a private network. If the internal_address is not set, the address is used for inter-host communication.

Per https://rancher.com/docs/rke/v0.1.x/en/config-options/nodes/#internal-address

You likely have a firewall issue.

Check your active zones and what interfaces are in those zones.

firewall-cmd --get-active-zones