Responses from kubernetes containers getting lost

I have installed kubernetes on openstack. The setup has one master and one node on coreos.

I have a pod hosting an SIP application on UDP port 5060 and I have created service as NODEPORT on 5060.

The spec:

"spec": {
    "ports": [
      {
        "port": 5061,
        "protocol": "UDP",
        "targetPort": 5060,
    "nodeport": 5060,
    "name": "sipu"
      }
    ],
    "selector": {
      "app": "opensips"
    },
    "type": "NodePort"
  }

IPs

Public IP of node: 192.168.144.29.
Private IP of node: 10.0.1.215..
IP of the container:10.244.2.4.
docker0 interface: 10.244.2.1.

Now, the problem. I send a SIP request to the application and expect a 200 OK response, which I am not getting.

To trace the same, I installed TCPDUMP on the container and the node. On the container, I can see the request and response packet captured while on the node itself I just see the request packet. Don't understand why the packet is getting lost.

Below is tcpdump of container:

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 1514 bytes
06:12:20.391171 IP (tos 0x0, ttl 64, id 13372, offset 0, flags [DF], proto UDP (17), length 1010)
    10.244.2.1.50867 > 10.244.2.4.5060: [bad udp cksum 0x1ddc -> 0xe738!] SIP, length: 982
        PUBLISH sip:service-1@opensipstest.org SIP/2.0
        Via: SIP/2.0/UDP 192.168.144.10:5060;branch=z9hG4bK-5672-1-0
        Max-Forwards: 20
        From: service <sip:service-1@opensipstest.org>;tag=1
        To: sut <sip:service-1@opensipstest.org>
        Call-ID: 1-5672@192.168.144.10
        CSeq: 1 PUBLISH
        Expires: 3600
        Event: presence
        Content-Length:   607
        User-Agent: Sipp v1.1-TLS, version 20061124

        <?xml version="1.0"?>
        <deleted presence xml to reduce size>

06:12:20.401126 IP (tos 0x10, ttl 64, id 11888, offset 0, flags [DF], proto UDP (17), length 427)
    10.244.2.4.5060 > 10.244.2.1.5060: [bad udp cksum 0x1b95 -> 0xeddc!] SIP, length: 399
        SIP/2.0 200 OK
        Via: SIP/2.0/UDP 192.168.144.10:5060;received=10.244.2.1;branch=z9hG4bK-5672-1-0
        From: service <sip:service-1@opensipstest.org>;tag=1
        To: sut <sip:service-1@opensipstest.org>;tag=332ff20b76febdd3c55f313f3efc6bb8-ca08
        Call-ID: 1-5672@192.168.144.10
        CSeq: 1 PUBLISH
        Expires: 3600
        SIP-ETag: a.1450478491.39.2.0
        Server: OpenSIPS (1.8.4-notls (x86_64/linux))
        Content-Length: 0


06:12:20.401364 IP (tos 0x0, ttl 64, id 13374, offset 0, flags [DF], proto UDP (17), length 427)
    10.244.2.1.58836 > 10.244.2.4.5060: [bad udp cksum 0x1b95 -> 0x1bcc!] SIP, length: 399
        SIP/2.0 200 OK
        Via: SIP/2.0/UDP 192.168.144.10:5060;received=10.244.2.1;branch=z9hG4bK-5672-1-0
        From: service <sip:service-1@opensipstest.org>;tag=1
        To: sut <sip:service-1@opensipstest.org>;tag=332ff20b76febdd3c55f313f3efc6bb8-ca08
        Call-ID: 1-5672@192.168.144.10
        CSeq: 1 PUBLISH
        Expires: 3600
        SIP-ETag: a.1450478491.39.2.0
        Server: OpenSIPS (1.8.4-notls (x86_64/linux))
        Content-Length: 0

And tcpdump of node:

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 1514 bytes
06:12:20.390772 IP (tos 0x0, ttl 127, id 20196, offset 0, flags [none], proto UDP (17), length 1010)
    192.168.144.10.5060 > 10.0.1.215.5060: [udp sum ok] SIP, length: 982
        PUBLISH sip:service-1@opensipstest.org SIP/2.0
        Via: SIP/2.0/UDP 192.168.144.10:5060;branch=z9hG4bK-5672-1-0
        Max-Forwards: 20
        From: service <sip:service-1@opensipstest.org>;tag=1
        To: sut <sip:service-1@opensipstest.org>
        Call-ID: 1-5672@192.168.144.10
        CSeq: 1 PUBLISH
        Expires: 3600
        Event: presence
        Content-Length:   607
        User-Agent: Sipp v1.1-TLS, version 20061124

        <?xml version="1.0"?>
        <presence xmlns="urn:ietf:params:xml:ns:pidf" />

Nodeport rules from Iptable

Chain KUBE-NODEPORT-CONTAINER (1 references)
 pkts bytes target     prot opt in     out     source               destination
   12  8622 REDIRECT   udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/opensips:sipu */ udp dpt:5060 redir ports 40482
    3    95 REDIRECT   udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/my-udp-service: */ udp dpt:6000 redir ports 47497

Chain KUBE-NODEPORT-HOST (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/opensips:sipu */ udp dpt:5060 to:10.0.1.215:40482
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/my-udp-service: */ udp dpt:6000 to:10.0.1.215:47497

I am happy to share more info if required. I have tried to dig in my capacity, but now I don't know what to look therefore requesting some help here.

EDIT

I did the same test on TCP. On TCP, it works as expected.

Thanks

5060 is outside the default Service Node Port Range: http://kubernetes.github.io/docs/user-guide/services/#type-nodeport

Creation of the service should have failed.

You can try a different port, or create a cluster using a different range, by specifying --service-node-port-range on kube-apiserver.

https://github.com/kubernetes/kubernetes/blob/43792754d89feb80edd846ba3b40297f2a105e14/cmd/kube-apiserver/app/options/options.go#L232

You should also check whether the response can be seen from other nodes. There are issues with "hairpin mode" when communicating within the same node.

Also, when reporting problems, please specify the Kubernetes release.