Behavior of python's select() with partial rec

Question

I have created a SSL socket (server side) and place the socket into a select() queue. The select() properly returns when the socket "is ready" to read.

I then recv(1024) bytes. Under some circumstances this will get all the data, in others it may not.

However, if there is still data in the socket buffer (because I didn't recv() it all), and I pass that same socket into select() again, it will not be returned as being "ready" to read even though I know there is data there.

I suppose my question is really to confirm what "ready to be read" really means from select()'s perspective and what the best way to handle this would be. Continuing to recv() until EWOULDBLOCK seems sort of hack-ish given that I'm using select().

Am I thinking about this incorrectly? I realize I could use a larger recv buffer, but there is always the possibility that there would be more to read than recv can pull -- so what is the "right" way to handle this coming out of a select()?

Thanks in advance.

EDIT: As noted in the comments, I neglected to mention that this is an SSL server and apparently select() behaves differently when using wrapped sockets.

Answer 1

select cares only about the readiness of a socket from the view of the OS kernel. That is checking if a socket is ready for recv with select only checks if there are data in the socket buffer. But with SSL it is different since user space buffering is involved.

Even if you read only a few bytes from a SSL socket it needs to read the full SSL record which contains the encrypted data, decrypt the full record and then it can return the few bytes you've requested. The rest of the data will be buffered in user space for the next read. But, the full SSL record is removed from the OS socket buffer which means that select might not show you that there are still data available.

There are two ways to work around this. One way is to use the pending method to find out of there are still data buffered in user space. The other is to always recv in huge chunks so that no data will be buffered in user space. Since the maximum size of a SSL record is 16k and each recv handles only a single SSL record (implementation detail in openssl SSL_read) it would work to always call recv with a size of at least 16384.