SSL_CONNECT fails with SSL_ERROR_SYSCALL error

Question

Having strange connection failure with openssl SSLConnect with my SSLCLIENT.

We are trying to establish ssl connection with a server. We notice that SSL_CONNECT is failing with error code "SSL_ERROR_SYSCALL".

For further depth we tried printing strerror(errno) which return "scuccess" "0".

However i am just trying to understand what might be the exact cause for this issue

Added code snippet for SSL init and connect::

request some guidance:

int setupSSL(int server){ 

        int retVal = 0; 
        int errorStatus = 0; 
        int retryMaxCount = 6; 
        static int sslInitContext=0; 

    if(sslInitContext == 0) 
    { 
                if(InitCTX() != 0) 
                { 
                        return -1; 
                } 
                else 
                { 
                        sslInitContext=1; 
                } 
    } 

    retVal = SSL_set_fd(ssl, server);    /* attach the socket descriptor */ 
        if ( retVal != 1 ) 
        {   
                /* perform the connection */ 
                sprintf(debugBuf,"SYSTEM:SOCKET:Could not set ssl FD: %d %s\n",retVal,strerror(retVal)); 
                debug_log(debugBuf,TRACE_LOG); 
                CloseSocket(server); 
                return -1; 
        } 
        do 
        { 
                retVal = SSL_connect(ssl); 
                errorStatus = SSL_get_error (ssl, retVal); 
                switch (errorStatus) 
                { 
                        case SSL_ERROR_NONE: 
                                retVal = 0;                         
                                break; 
                        case SSL_ERROR_WANT_READ: 
                        case SSL_ERROR_WANT_WRITE: 
                                retVal = 1; 
                                break; 
                        default: 
                                sprintf(debugBuf,"SYSTEM:SSL_SOCKET:Could not build SSL session(Other error): %d %s\n",errorStatus,strerror(errno)); 
                                debug_log(debugBuf,TRACE_LOG); 
                                CloseSocket(server); 
                                return -1;                             
                } 
                sprintf(debugBuf,"SYSTEM:SSL_SOCKET:SSL CONNECTION Under PROGRESS: %d with remaining retries %d\n",errorStatus,retryMaxCount); 
                debug_log(debugBuf,TRACE_LOG); 

                if (retVal) 
                { 
                        struct timeval tv;         
                        fd_set sockReadSet; 
                        tv.tv_sec = 2; 
                        tv.tv_usec = 0;          
                        FD_ZERO(&sockReadSet); 
                        FD_CLR(server, &sockReadSet); 
                        FD_SET(server,&sockReadSet); 

                        retVal = select(server+1, &sockReadSet, NULL, NULL, &tv); 
                        if (retVal >= 1) 
                        { 
                                  retVal = 1; 
                        } 
                         else 
                        { 
                                  retVal = -1; 
                        } 
                        retryMaxCount--; 
                        if (retryMaxCount <= 0 ) 
                                break;         
                } 
        }while(!SSL_is_init_finished (ssl) && retVal == 1); 
    cert = SSL_get_peer_certificate(ssl); 
    if(cert == NULL) 
        { 
            debug_log("SYSTEM:SSL_SOCKET:Unable to retrive server certificate\n",TRACE_LOG); 
                CloseSocket(server); 
                return -1; 
    } 
    if(SSL_get_verify_result(ssl)!=X509_V_OK) 
        { 
            debug_log("SYSTEM:SSL_SOCKET:Certificate doesn't verify\n",TRACE_LOG); 
                CloseSocket(server); 
            return -1; 
    } 
    /*X509_NAME_get_text_by_NID (X509_get_subject_name (cert),  NID_commonName,  peer_CN, 256); 
    if(strcasecmp(peer_CN, cnName)){ 
         debug_log("SYSTEM:SSL_SOCKET:Common name doesn't match host name\n",TRACE_LOG); 
         return -1; 
     }*/ 
           return 0; 
   // LoadCertificates(ctx, CertFile, KeyFile); 
} 


int InitCTX(void) 
{ 
        int errorStatus = 0; 
        static int isSslInit = 1; 
        if(isSslInit) 
        { 
                OpenSSL_add_all_algorithms();/* Load cryptos, et.al. */ 
                SSL_load_error_strings();/* Bring in and register error messages */ 
                if(SSL_library_init() < 0) 
                { 
                        debug_log("SYSTEM:SSL_SOCKET:Could not initialize the OpenSSL library\n",TRACE_LOG); 
                        return -1; 
                } 
                method = TLSv1_client_method(); 
                isSslInit=0; 
        } 
    ctx = SSL_CTX_new(method);/* Create new context */ 
    if ( ctx == NULL) 
        { 
            debug_log("SYSTEM:SSL_SOCKET:Unable to create a new SSL context structure\n",TRACE_LOG); 

            //sprintf(debugBuf,"SYSTEM:SSL_SOCKET:Unable to create a new SSL context structure: %d %s\n",errorStatus,strerror(retVal)); 
                //debug_log(debugBuf,TRACE_LOG); 

            return -1; 
    } 
    SSL_CTX_set_options(ctx, SSL_OP_NO_SSLv2); 
    if (SSL_CTX_use_certificate_file(ctx,CertFile, SSL_FILETYPE_PEM) <= 0) 
        { 
                SSL_CTX_free(ctx); 
                ctx = NULL; 
                 debug_log("SYSTEM:SSL_SOCKET:Error setting the certificate file.\n",TRACE_LOG); 
                 return -1; 
    } 

        /* Set the list of trusted CAs based on the file and/or directory provided*/ 
          if(SSL_CTX_load_verify_locations(ctx,CertFile,NULL)<1) 
        { 
                SSL_CTX_free(ctx); 
                ctx = NULL; 
                  debug_log("SYSTEM:SSL_SOCKET:Error setting verify location.\n",TRACE_LOG); 
                  return -1; 
          } 
         SSL_CTX_set_verify(ctx,SSL_VERIFY_PEER,NULL); 
          SSL_CTX_set_timeout (ctx, 300); 


        ssl = SSL_new(ctx);      /* create new SSL connection state */ 
        if(ssl == NULL) 
        { 
                sprintf(debugBuf,"SYSTEM:SOCKET:SSL:Unable to create SSL_new context\n"); 
                debug_log(debugBuf,DEBUG_LOG); 
                if(ctx != NULL) 
                  SSL_CTX_free(ctx); 
                return -1; 
        } 

          return 0; 
} 

Also is it advised to maintain SSL context for new connections or should we destroy and re init the ssl context??

Added PCAP info:

https://drive.google.com/file/d/0B60pejPe6yiSUk1MMmI1cERMaFU/view?usp=sharing

client: 198.168.51.10 (198.168.51.10), Server: 192.168.96.7 (192.168.96.7)

Answer 1

We are trying to establish ssl connection with a server. We notice that SSL_CONNECT is failing with error code "SSL_ERROR_SYSCALL".

This is usually the case if the other side is simply closing the connection. Microsoft SChannel does this on many kind of handshake problems instead of sending a TLS alert back. This can happen for problems like invalid protocol or no common ciphers etc. It also can happen if you try to do a TLS handshake with a server which does not speak TLS at all on this port. Look at logs at the server side for problems.

Of course it can also be something different so you might check the errno to get more details about the problem. It might also help if you do a packet capture to check what's going on on the wire. Best would be to do this capture on the client and server side to make sure that no middlebox like a firewall is tampering with the connection.

Also is it advised to maintain SSL context for new connections or should we destroy and re init the ssl context??

The context is just a collection of settings, certificates etc and is not affected by the SSL connection itself. You can reuse it for other connection later or at the same time.

EDIT, after the packet capture was attached:

There are multiple TCP connection in the file between client and server and only inside a single one the client tries to initiate a handshake, i.e. the ClientHello can be seen. The server closes the connection. A few things a interesting:

• TCP handshake takes very long. The server only replies after 1.6 seconds after receiving the SYN with the SYN+ACK. Also the other replies take 800ms which is very long given that both addresses are in a private network (192.168.0.0). This might indicate a slow connection or VPN (this is about the latency of a satellite link), some middlebox (firewall) slowing everything down or a really slow server.
• Client sends TLS 1.0 request. It might be that the server will do only TLS 1.1+. Some TLS stacks (see above) simply close the connection on such errors instead of sending an unsupported protocol alert. But given that the server is slow it might also be old and only support SSL 3.0 or lower.
• Client does not use SNI extension. More and more servers need this and might simply close if they don't get the extension.

It is hard to know what really is going on without having access to the server. I recommend to look for error messages on the server side and use tools like SSLyze to check the requirements of the server, i.e. supported TLS versions, ciphers etc.

Apart from that client offers dangerously weak ciphers like various EXPORT ciphers. This looks for me like the defaults of a considerably old OpenSSL version.