Port not accessible even after being exposed. Conn

Question

we created a docker container like this:

docker container create \
    --name orderer \
    --network dscsa_net \
    --workdir $WORK_DIR \
    --expose=7050 \
    hyperledger/fabric-orderer:1.3.0 ./start-orderer.sh

but are unable to connect to port 7050 on the container.

root@dcee7e74266f:/home# nc -vz 10.0.0.194 7050
nc: connect to 10.0.0.194 port 7050 (tcp) failed: Connection refused

we are able to ping the container:

root@dcee7e74266f:/home# ping 10.0.0.194
PING 10.0.0.194 (10.0.0.194) 56(84) bytes of data.
64 bytes from 10.0.0.194: icmp_seq=1 ttl=64 time=0.810 ms
64 bytes from 10.0.0.194: icmp_seq=2 ttl=64 time=1.30 ms
64 bytes from 10.0.0.194: icmp_seq=3 ttl=64 time=0.668 ms
64 bytes from 10.0.0.194: icmp_seq=4 ttl=64 time=1.10 ms
64 bytes from 10.0.0.194: icmp_seq=5 ttl=64 time=0.631 ms
^C
--- 10.0.0.194 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 0.631/0.902/1.301/0.261 ms

and also see a process listening on port 7050 on the container:

root@9756199efefa:/home# netstat -tuplen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
tcp        0      0 127.0.0.1:7050          0.0.0.0:*               LISTEN      0          10097930    7/orderer       
tcp        0      0 127.0.0.11:34865        0.0.0.0:*               LISTEN      0          10097705    -               
udp        0      0 127.0.0.11:51385        0.0.0.0:*                           0          10097704    -  

what is going on here? how can we fix this?

EDIT: we are on a overlay network. The publish flag suggested in the answer is n/a as we are doing container to container communication. Anyway we tried it and it doesn't work.

There is one thing we have noticed which is if we run:

docker network inspect <our-network-name> 

Among other things, it prints out a containers section but in that section only the containers on the host from which docker network inspect is executed are listed. The containers hosted on other nodes are not listed (also mentioned here).

we verified that if we run:

docker node ls

all the nodes are part of the swarm.

It seems other people have also run into this issue e.g., here but what is the solution?

Note: we are able to connect to another container running a different service exposed on port 7054. This container was created without even using the expose flag.

root@dcee7e74266f:/home# nc -zv 10.0.0.164 7054
Connection to 10.0.0.164 7054 port [tcp/*] succeeded!

Did further debugging with tcpdump and output of tcpdump is identical to the output when someone tries to connect to a port on which no process is listening. But as shown earlier netstat shows a process that is listening and we can connect to the process from localhost.

Output of tcpdump:

root@dcee7e74266f:/test# tcpdump -s0 host 10.0.0.195 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
23:44:45.978583 IP dcee7e74266f.52148 > orderer.dscsa_net.7050: Flags [S], seq 3845506108, win 28200, options [mss 1410,sackOK,TS val 4203049443 ecr 0,nop,wscale 7], length 0
23:44:45.979324 IP orderer.dscsa_net.7050 > dcee7e74266f.52148: Flags [R.], seq 0, ack 3845506109, win 0, length 0

The R flag tells client to reset the connection.

Output of traceroute:

root@dcee7e74266f:/test# traceroute 10.0.0.195     
traceroute to 10.0.0.195 (10.0.0.195), 30 hops max, 60 byte packets
 1  orderer.dscsa_net (10.0.0.195)  1.008 ms  0.900 ms  0.872 ms

Answer 1

Expose only sets metadata on the image or container, it does not make the port externally accessible. The option you are looking for is publish:

docker container create \
    --name orderer \
    --network dscsa_net \
    --workdir $WORK_DIR \
    --publish=7050:7050 \
    hyperledger/fabric-orderer:1.3.0 ./start-orderer.sh

Answer 2

Solved this issue thanks to 1. The server listening to 127.0.0.1 was the problem. Once we changed the listening address to 0.0.0.0 (shows as ::: in netstat output below), we are able to connect to the server:

root@e9766a94d102:/home# netstat -tuplen          
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
tcp        0      0 127.0.0.11:37641        0.0.0.0:*               LISTEN      0          12821468    -               
tcp6       0      0 :::7050                 :::*                    LISTEN      0          12821696    7/orderer       
udp        0      0 127.0.0.11:51855        0.0.0.0:*                           0          12821467    -       

there is no need for either expose or publish flags. note to self: wasted 1.5 days on this.