Can TCP SOCKET handles be set not inheritable?

Question

Windows handles can be set to be either inheritable or not, to control whether child processes will receive them (when bInheritHandles in CreateProcess is TRUE). However, using SetHandleInformation to mark a SOCKET non-inheritable does not always work. In particular, when certain Layered Service Providers (LSPs) are installed, the child process inherits the handle anyway. This is particularly likely to cause bugs with listening sockets. (But, because of another issue, if the child were to try using the socket, it would not be able! A real catch-22!)

Steps to reproduce

1. Create eg a listening socket. Mark it non-inheritable using SetHandleInformation.
2. Spawn a child, with bInheritHandles true.
3. Close the socket in the parent, and attempt to re-bind to the port.

When a (non-IFS) LSP is installed, eg. PCTools Internet Security, the listening socket will be open in the child (visible in netstat), despite SetHandleInformation being called on the socket to disable inheritance before creating the child.

For an alternative scenario, see the (brief) steps in KB2398202.

What workarounds are available?

Answer 1

Short answer

It is not possible in general to set SOCKET handles non-inheritable. That is, when certain (non-IFS) LSPs are installed, even if you mark handles in your process specifically non-inheritable, it is not possible to stop a child process with bInheritHandles=TRUE from receiving them.

Explanation

LSPs are commonly used by firewall or A/V products to filter all TCP connections. The LSP is a DLL loaded into your process by WinSock which handles all the TCP operations, typically by performing some filtering and then passing the call straight down to the underlying WinSock implementation. The LSP works by creating a dummy handle for each actual SOCKET handle produced the WinSock implementation: your call to WSASocket will give you the dummy handle; when you use the dummy handle, the call is sent to the LSP which created it; the LSP then maps the dummy back to the actual handle, and passes on the operation (such as accept or bind) to the underlying handle.

The problem is therefore that calling SetHandleInformation on the sockets you create is not enough: the underlying handle that you never see (used internally by the LSP) is still inherited by child processes.

Workarounds

1. Don't ever call CreateProcess allowing any inheritance from an application which uses sockets. This is the most reliable solution. Instead, to set up a communication with the child, create a named pipe with suitable permissions, pass its name on the commandline to the child, and connect back in the child. Then manually pass over any handles you wish the child to inherit. This is secure, since although the commandline may be readable by other users, only the actual user token of the child can connect to the pipe if it's set up correctly.
   This is extremely inelegant if all you want to do is something simple like redirect a child's stdio, since you have to control the argument parsing in the child. To work around that, create a wrapper binary which reads the named pipe name from the commandline and connects, sets the handle inheritable, and reinvokes the remaining arguments with stdio redirected. It is safe to inherit handles from the wrapper, because there are no sockets in the process.
2. Alternatively, on Vista (with KB2398202), Windows 7 (with SP1), and later, the WSA_FLAG_NO_HANDLE_INHERIT flag was added to WSASocket. (This is as much documentation of the problem as I can find from Microsoft. Creating the hotfix is pretty much acknowledgement that it's not possible without it to prevent the Base Service Provider handles from being inherited. It's not well advertised though!)
3. Finally, on Vista, there are also ioctls which allow the handles used by a Base Service Provider to be queried. These can then be marked non-inheritable. This is a pain though, and still doesn't fix things for XP.