Nginx TCP forwarding based on hostname

Question

With the release of TCP load balancing for the Nginx community version, I would like to mix OpenVPN and SSL pass-through data. The only way for Nginx to know how to route the traffic is via their domain name.

 vpn1.app.com ─┬─► nginx at 10.0.0.1 ─┬─► vpn1  at 10.0.0.3
 vpn2.app.com ─┤                      ├─► vpn2  at 10.0.0.4
https.app.com ─┘                      └─► https at 10.0.0.5

I have taken a look at the TCP guides and the module documentation, but it doesn't seem well referenced. If anyone can point me to the right direction, i'd be grateful.

Answer 1

Assumptions

If I understand you correctly, you effectively want nginx to listen at a single IP address and TCP port combination (e.g., listen 10.0.0.1:443), and then, depending on the characteristic of the incoming TCP stream traffic, route it to one of the 3 different IP addresses.

You don't explicitly mention how you expect it to differentiate between the 3 different domains at stake, but my assumption is that you assume it's all just TLS, and must want to employ some sort of a TLS SNI (Server Name Indication) mechanism for domain-based differentiation.

I would believe that the stream-related documentation provided at http://nginx.org/docs/ is quite authoritative and exhaustive for the modules at stake (I'm listing all of it here, since apparently there's no central place for cross-referencing this yet, e.g., no references from the "stream core" module to the submodules yet (and docs/stream/ just redirects back docs/), which is indeed quite confusing, since stuff like http://nginx.org/r/upstream is only documented to apply to http, without any mention of applicability to stream, even if the directives are about the same in the end):

• http://nginx.org/docs/stream/ngx_stream_core_module.html
• http://nginx.org/docs/stream/ngx_stream_access_module.html
• http://nginx.org/docs/stream/ngx_stream_limit_conn_module.html
• http://nginx.org/docs/stream/ngx_stream_proxy_module.html
• http://nginx.org/docs/stream/ngx_stream_ssl_module.html
• http://nginx.org/docs/stream/ngx_stream_upstream_module.html

---

Answer

Note that each nginx directive, from each module, has a limited number of applicable Context's.

As such, unfortunately, there is simply no directive to snoop into SNI here!

To the contrary, it's actually documented in stream_core that, to quote, "Different servers must listen on different address:port pairs.", which, as you may note, is also contrary to how the listen directive works within the more-common http_core, and is a rather unambiguous reference to the fact that no kind of SNI support is presently implemented for the listen within stream.

---

Discussion

As a discussion point and a resolution suggestion, the assumption that OpenVPN traffic is just TLS with the snoopable SNI is also not necessarily correct (but I'm not too familiar with OpenSSL or SNI):

• Consider that even if SNI is passively snoopable today, that's clearly contrary to the promise of TLS of keeping the connection secure, and, as such, may change in a future version of TLS.

• For the sake of discussion, if OpenVPN is just using a TLS connection, and if it is NOT using TLS for authenticating users with user certificates (which would make it much more difficult to MitM the stream, yet still carry the authentication data all along), then, theoretically, if nginx did have SNI support around the listen within stream, then you'd possibly have been able to actively MitM it with nginx (since proxy_ssl is already supported in stream_proxy).

Most importantly, I believe OpenVPN may best be run over its own UDP-based protocol, in which case, you can use the same IP address and port number for one instance of the TCP-based https and another one of the UDP-based OpenVPN without a conflict.

In the end, you may ask, what would the stream module be useful for anyways, then? I believe its target audience would be, (0), load balancing HTTP/2 with multiple upstream servers, based on the hash of the IP-address of the client, for example, and/or, (1), a more straightforward and protocol-agnostic replacement for stunnel.

Answer 2

This is now possible with the addition of the ngx_stream_ssl_preread module added in Nginx 1.11.5 and the ngx_stream_map module added in 1.11.2.

This allows Nginx to read the TLS Client Hello and decide based on the SNI extension which backend to use.

stream {

    map $ssl_preread_server_name $name {
        vpn1.app.com vpn1_backend;
        vpn2.app.com vpn2_backend;
        https.app.com https_backend;
        default https_default_backend;
    }

    upstream vpn1_backend {
        server 10.0.0.3:443;
    }

    upstream vpn2_backend {
        server 10.0.0.4:443;
    }

    upstream https_backend {
        server 10.0.0.5:443;
    }

    upstream https_default_backend {
        server 127.0.0.1:443;
    }

    server {
        listen 10.0.0.1:443;
        proxy_pass $name;
        ssl_preread on;
    }
}

Answer 3

AS @Lochnair mentioned, you can use ngx_stream_map module and variable $server_addr to resolve this problem. Here is my example.

My host IP is 192.168.168.22, and I use keepalived bound 2 virtual IP to eth0.

$sudo ip a
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 5c:f3:fc:b9:f0:84 brd ff:ff:ff:ff:ff:ff
inet 192.168.168.22/24 brd 192.168.168.255 scope global eth0
   valid_lft forever preferred_lft forever
inet 192.168.168.238/32 scope global eth0
   valid_lft forever preferred_lft forever
inet 192.168.168.239/32 scope global eth0
   valid_lft forever preferred_lft forever

$nginx -v
nginx version: nginx/1.13.2

$cat /etc/nginx/nginx.conf
...
stream {
    upstream pod53{
        server 10.1.5.3:3306;
    }
    upstream pod54{
        server 10.1.5.4:3306;
    }

    map $server_addr $x {
        192.168.168.238 pod53;
        192.168.168.239 pod54;
    }
    server {
        listen 3306;
        proxy_pass $x;
    }
}

Thus, I can visit different MySQL service with the same port 3306 via different VIPs. Just like visiting different HTTP service with the same port via diffrent server_name.

192.168.168.238 -> 10.1.5.3
192.168.168.239 -> 10.1.5.4