Linux Socket Input Notification on a Client-Server

Question

This question is a followup to this question and answer

I'm building a minimalistic remote access programm on Linux using SSL and socket programming.

The problem arose in the following protocol chain

1. Client sends command
2. Server recieves it
3. Server spawns a child, with input, output and error dup-ed with the server-client socket (so the input and output would flow directly though the sockets)
4. Server waits for the child and waits for a new command

When using SSL, you cannot use read and write operations directly, meaning the child using SSL sockets with send plain data (because it won't use SSL_write or SSL_read, but the client will, and this will create problems).

So, as you could read from the answer, one solution would be to create 3 additional sets of local sockets, that only server and it's child will share, so the data could flow unencrypted, and only then send it to the client with a proper SSL command.

So the question is - how do I even know when a child wants to read, so I could ask for input from the client. Or how do I know when the child outputs something so I could forward that to the client

I suppose there should be created some threads, that will monitor and put locks on the SSL structure to keep the order, but I still can't imagine how the server would get notified, when the child application hit a scanf("%d") or something else.

Answer 1

To illustrate what need to be done use the following Python program. I've used Python only because it is easy to read but the same can be done in C, only with more lines of code and harder to read.

Let's first do some initialization, i.e. create some socket server, SSL context, accept a new client, wrap the client fd into an SSL socket and do some initial communication between client and socket server. Based on your previous question you probably know how to this in C already and the Python code is not that far away from what you do in C:

import socket
import ssl
import select
import os

local_addr = ('',8888) # where we listen
cmd = ['./cmd.pl']  # do some command reading stdin, writing stdout

ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
ctx.load_cert_chain('server_cert_and_key.pem')

srv = socket.socket()
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind(local_addr)
srv.listen(10)

try:
    cl,addr = srv.accept()
except:
    pass
cl = ctx.wrap_socket(cl,server_side=True)
print("new connection from {}".format(addr))

buf = cl.read(1024)
print("received '{}'".format(buf))
cl.write("hi!\n")

After this setup is done and we have a SSL connection to the client we will fork the program. This program will read on stdin and write to stdout and we want to forward the decrypted input from the SSL client as input to the program and forward the output of the program encrypted to the SSL client. To do this we create a socketpair for exchanging data between parent process and forked program and remap the one side of the socketpair to stdin/stdout in the forked child before execv the program. This works also very similar to C:

print("forking subprocess")
tokid, toparent = socket.socketpair()
pid = os.fork()
if pid == 0:
    tokid.close()
    # child: remap stdin/stdout and exec cmd
    os.dup2(toparent.fileno(),0)
    os.dup2(toparent.fileno(),1)
    toparent.close()
    os.execv(cmd[0],cmd)
    # will not return

# parent process
toparent.close()

Now we need to read the data from SSL client and forked command and forward it to the other side. While one might probably do this with blocking reads inside threads I prefer it event based using select. The syntax for select in Python is a bit different (i.e. simpler) than in C but the idea is exactly the same: the way we call it it will return when we have data from either client or forked command or if one second of no data has elapsed:

# parent: wait for data from client and subprocess and forward these to
# subprocess and client
done = False
while not done:
    readable,_,_ = select.select([ cl,tokid ], [], [], 1)
    if not readable:
        print("no data for one second")
        continue

Since readable is not empty we have new data waiting for us to read. In Python we use recv on the file handle, in C we would need to use SSL_read on the SSL socket and recv or read on the plain socket (from socketpair). After the data is read we write it to the other side. In Python we can use sendall, in C we would need to use SSL_write on the SSL socket and send or write on the plain socket - and we would also need to make sure that all data were written, i.e. maybe try multiple times.

There is one notable thing when using select in connection with SSL sockets. If you SSL_read less than the maximum SSL frame size it might be that the payload of the SSL frame was larger than what was requested within SSL_read. In this case the remaining data will be buffered internally by OpenSSL and the next call to select might not show more available data even though there are the already buffered data. To work around this one either needs to check with SSL_pending for buffered data or just use SSL_read always with the maximum SSL frame size:

    for fd in readable:
        # Always try to read 16k since this is the maximum size for an
        # SSL frame. With lower read sizes we would need to explicitly
        # deal with pending data from SSL (man SSL_pending)
        buf = fd.recv(16384)

        print("got {} bytes from {}".format(len(buf),"client" if fd == cl else "subprocess"))
        writeto = tokid if fd == cl else cl
        if buf == '':
            # eof
            writeto.close()
            done = True
            break # return from program
        else:
            writeto.sendall(buf)

print("connection done")

And that's it. The full program is also available here and the small program I've used to test is available here.