SSL_accept with blocking socket

Question

I made a server with SSL and blocking sockets. When I connect with telnet (so it does not do the handshake), the SSL_accept blocks indefinitely and blocks every new handshake/accept (and by definition new connections).

How can I solve this awful problem ?

Answer 1

Don't connect with telnet?

Normally, a service that can use either TLS or a plaintext connections works by establishing the connection in plaintext, then offering a command to request an "upgrade" of the connection to use TLS. Extended SMTP and the "STARTTLS" command is an example of this.

Services without such a command usually use distinct ports for TLS and non-TLS traffic. For example, HTTP on port 80 and HTTPS on port 443. A plaintext connection to port 443 isn't going to work.

What behavior would you like to see?

Answer 2

You can put the socket in non-blocking mode, then you'll get case SSL_ERROR_WANT_READ, or SSL_ERROR_WANT_WRITE from SSL_accept. You can then sleep a little, and try SSL_accept again. After some timeout value, you can quit and close the ssl and socket handles.

Note that this will affect all SSL operations, which means you'll need to do a similiar loop for all your read/write/shutdown calls. Basically, any call that can return WANT_READ or WANT_WRITE.

If you don't like the idea of polling, you can use select to figure out if you have data available on the socket...but that can get a bit complicated.

You can also try putting the socket back into blocking mode after the SSL_accept loop, and continue with your application.

Answer 3

Why not just set the socket stream to non-blocking mode before calling SSL_accept(), and then block on something like select() with a timeout if SSL_accept() returns SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE? Alternatively, you can block on select() before calling SSL_accept(). Either should work. That way you can at least bound the time the connection is blocked due to the DoS like behavior/attack.

Bear in mind that SSL/TLS is record-oriented, meaning you must loop until the full record is read. SSL_pending() can help in such cases.

Answer 4

I think code below might help others to solve the issue. It is not fully tested take it as inspiration.

  //Nonblocking SSL accept based on ACE/ace/SSL/SSL_SOCK_Acceptor.cpp
  SSL_CTX* ctx;
  ctx = initServerCTX(); // initialize SSL
  loadCertificates(ctx, certificate, privateKey); // load certs

  ...

  SSL* ssl = SSL_new(ctx); /* get new SSL state with context */
  SSL_set_fd(ssl, fd); /* set connection socket to SSL state */

  int flags = fcntl(fd, F_GETFL, 0);
  if (flags < 0)
  {
    printf("fcntl: F_GETFL \n");
    return false;
  }
  if (fcntl(fd, F_SETFL, flags | O_NONBLOCK) < 0)
  {
    printf("fcntl: F_SETFL \n");
    return false;
  }

  int status = -1;
  struct timeval tv, tvRestore;
  tv.tv_sec = 2;
  tv.tv_usec = 0;
  tvRestore = tv;

  fd_set writeFdSet;
  fd_set readFdSet;

  do
  {
    tv = tvRestore;
    FD_ZERO(&writeFdSet);
    FD_ZERO(&readFdSet);

    status = ::SSL_accept(ssl);
    switch (::SSL_get_error(ssl, status))
    {
    case SSL_ERROR_NONE:
      status = 0; // To tell caller about success
      break; // Done

    case SSL_ERROR_WANT_WRITE:
      FD_SET(fd, &writeFdSet);
      status = 1; // Wait for more activity
      break;

    case SSL_ERROR_WANT_READ:
      FD_SET(fd, &readFdSet);
      status = 1; // Wait for more activity
      break;

    case SSL_ERROR_ZERO_RETURN:
    case SSL_ERROR_SYSCALL:
      // The peer has notified us that it is shutting down via
      // the SSL "close_notify" message so we need to
      // shutdown, too.
      printf("Peer closed connection during SSL handshake,status:%d", status);
      status = -1;
      break;
    default:
      printf("Unexpected error during SSL handshake,status:%d", status);
      status = -1;
      break;
    }

    if (status == 1)
    {
      // Must have at least one handle to wait for at this point.
      status = select(fd + 1, &readFdSet, &writeFdSet, NULL, &tv);

      // 0 is timeout, so we're done.
      // -1 is error, so we're done.
      // Could be both handles set (same handle in both masks) so
      // set to 1.
      if (status >= 1)
      {
        status = 1;
      }
      else // Timeout or failure
      {
        printf("SSL handshake - peer timeout or failure");
        status = -1;
      }
    }

  }
  while (status == 1 && !SSL_is_init_finished(ssl));

  flags = fcntl(fd, F_GETFL, 0);
  if (flags < 0)
  {
    printf("fcntl: F_GETFL \n");
    return false;
  }
  if (fcntl(fd, F_SETFL, flags & (~O_NONBLOCK)) < 0)
  {
    printf("fcntl: F_SETFL \n");
    return false;
  }


  return (status >= 0);