I know it is quite normal setting tcp_max_tw_buckets to a relatively small number such as 30000 or 50000, to avoid the situation when a host have a lots of time-wait state connections and application failed to open new one. It is something mentioned quite a lots. such as the question like this: How to reduce number of sockets in TIME_WAIT?
As before I know time-wait is a state to avoid TCP packets out of order, and it may be better using some other approach to coping it. And if you setting it to a small number thing may went wrong.
I feel I'm stucking at somewhere that I have to set tcp_max_tw_buckets to a small number, and don't know the specific scenarios I shall avoid it.
So my question is what is the side effect of setting tcp_max_tw_buckets to a very small value, and can I setup a specific scenario using lab environment, that a small number of tcp_max_tw_buckets will cause the trouble?
As you can see in this Kernel source, that option prevents graceful termination of the socket. In terms of the socket state, you have reduced the time wait duration for this connection to zero.
So what happens next? First off, you'll see the error message on your server. The rest is then a race condition for subsequent connections from your clients. Section 2 of rfc 1337 then covers what you may see. In short, some connections may show the following symptoms.
- Corruption of your data stream (because the socket accepts an old transmission).
- Infinite ACK loops (due to an old duplicate ACK being picked up).
- Dropped connections (due to old data turning up in the SYN-SENT state).
However, proving this may be hard. As noted in the same RFC:
The three hazards H1, H2, and H3 have been demonstrated on a stock Sun OS 4.1.1 TCP running in an simulated environment that massively duplicates segments. This environment is far more hazardous than most real TCP's must cope with, and the conditions were carefully tuned to create the necessary conditions for the failures.
The real answer to your question is that the correct way to avoid TIME_WAIT states is to be the end that receives the first close.
In the case of a server, that means that after you've sent the response you should loop waiting for another request on the same socket, with a read timeout of course, so that it is normally the client end which will close first. That way the TIME_WAIT state occurs at the client, where it is fairly harmless, as clients don't have lots of outbound connections.
