How to reproduce a silently dropped TCP/IP connect

Question

I have a situation where a Java program establishes a long-running TCP/IP connection to a server and said connection is functioning exactly as documented, except for one thing in an environment outside my control.

Every Saturday that server is rebooted, but for some reason this is not communicated properly to my client so the connection just hangs waiting for a response for ever and ever. This is exactly the same behaviour seen with stale JDBC-connections, where some router in between server and client has found the connection to be idle and dropped it without notification.

I need to be able to reproduce this scenario to be able to create a proper fix.

Hence I need to know if there is a good way to simulate a router silently dropping connections under my control?

I can put a Linux box or a Mac between my development environment and the server. I can run the JVM in a virtual machine (vmware player) if I can do the dropping there.

Suggestions are very welcome.

---

EDIT 2012-12-18: The solution that worked for me was to use a Java-based SOCKS5-proxy locally which could be suspended with Ctrl-Z at will, and tell my application to go through it.

Answer 1

A native Linux based solution is to use tc with netem. Netem is a special queue that can be attached to various network interfaces to cause delay, reordering and loss (see the web page for details). Some examples of netem use include:

tc qdisc change dev eth0 root netem loss 0.1%

To make eth0 drop 0.1% of all outgoing packets, and

tc qdisc change dev eth0 root netem loss 0.3% 25%

To create bursts of packet loss:

This will cause 0.3% of packets to be lost, and each successive probability depends by a quarter on the last one.

If you need to drop incoming packets as well, you can use the ifb (Intermediate Functional Block device). You set it up, forward all packets received by eth0 to the ifb, and attach netem to the ifb to create loss or delay. See ifb's documentation for more details.

Answer 2

Unplug the network cable from your computer.

Bring up a simple test of just this module. Connect to the server and get to the waiting part. Then, pull out your network plug.

It's important, of course, that you isolate this module and bring up only this module when you run this test. There may be other aspects of your program that require network access, and these should not be running.

Answer 3

Use Socat to forward your connection (it's very easy to set up as a TCP forwarder/proxy) even on the same machine if you want (by introducing a new port for the proxy to operate on and pointing it to the 'official' endpoint)

Check out the simple TCP forwarder example

Then when you want to test outages you can either kill it (and the socket will be closed) or potentially try stopping the process (CTRL Z linux) to see how your destination handles stalled connections.

Any questions? I'll try to answer... Good luck!

Answer 4

You should be able to temporarily apply a rule using iptables to drop all packets coming from the remote server (a.b.c.d). Something along the lines of :

iptables -A INPUT -s a.b.c.d -j DROP

When you want to turn the filter off

iptables -D INPUT -s a.b.c.d -j DROP

You can play with those rules to drop other specific packets as well, but I think this will accurately model what your client software would see if the server simply disappeared.

Note though, that when you turn the connection back on, the remote server may still have the old connection alive, so this won't perfectly simulate a reboot.