CFNetwork SSLHandshake failed (-9807) on iOS simul

Question

I'm trying to connect openssl s_server and iOS client using TCP SSL, but get CFNetwork SSLHandshake failed (-9807).

Objective-C code(from here):

- (void)viewDidLoad {
    [super viewDidLoad];

    printf("method started \n");

    NSBundle *bundle = [NSBundle bundleForClass:[self class]];
    NSData *iosTrustedCertDerData =
    [NSData dataWithContentsOfFile:[bundle pathForResource:@"servercert"
                                                    ofType:@"der"]];

    OSStatus            err = noErr;
    SecCertificateRef   cert;

    cert = SecCertificateCreateWithData(NULL, (CFDataRef) iosTrustedCertDerData);
    assert(cert != NULL);

    CFTypeRef result;

    NSDictionary* dict = [NSDictionary dictionaryWithObjectsAndKeys:
                          (id)kSecClassCertificate, kSecClass,
                          cert, kSecValueRef,
                          nil];

    err = SecItemAdd((CFDictionaryRef)dict, &result);
    assert(err == noErr || err == errSecDuplicateItem);

    printf("adding finished \n");


    if ((err == noErr) ||
        (err == errSecDuplicateItem)) {

        printf("success \n");

        CFReadStreamRef readStream;
        CFWriteStreamRef writeStream;
        CFStreamCreatePairWithSocketToHost(NULL,
                                           (CFStringRef)@"localhost",
                                           1678,
                                           &readStream,
                                           &writeStream);
        CFReadStreamSetProperty(readStream,
                                kCFStreamPropertySocketSecurityLevel,
                                kCFStreamSocketSecurityLevelTLSv1);
        CFReadStreamOpen(readStream);
        CFWriteStreamOpen(writeStream);

        UInt8 buf[] = "Hello from iOS";
        int bytesWritten = CFWriteStreamWrite(writeStream, buf, strlen((char*)buf));

    } else {
        printf("error!");
    }

    CFRelease(cert);
}

Sequence of commands to create the certificate and start the server (please pay attention to the 11 step: on the server side I use pem certificate and on the client side I use der certificate), file samples I got from this answer:

1.touch openssl-ca.cnf

2.Copy-pasted sample in openssl-ca.cnf. Only one line changed:

commonName_default = localhost

3.openssl req -x509 -config openssl-ca.cnf -newkey rsa:4096 -sha256 -nodes -out cacert.pem -outform PEM

4.touch openssl-server.cnf

5.Copy-pasted sample in openssl-server.cnf. Only two lines changed:

commonName_default = localhost

DNS.1 = localhost

6.openssl req -config openssl-server.cnf -newkey rsa:2048 -sha256 -nodes -out servercert.csr -outform PEM

7.Added 2 +1 sections in openssl-ca.cnf:

[ CA_default ]
    ...
    base_dir    = .
    certificate = $base_dir/cacert.pem  # The CA certifcate
    private_key = $base_dir/cakey.pem   # The CA private key
    new_certs_dir   = $base_dir     # Location for new certs after signing
    database    = $base_dir/index.txt   # Database index file
    serial      = $base_dir/serial.txt  # The current serial number

    unique_subject  = no            # Set to 'no' to allow creation of
                    # several certificates with same subject.

...

####################################################################
[ signing_policy ]
countryName     = optional
stateOrProvinceName = optional
localityName        = optional
organizationName    = optional
organizationalUnitName  = optional
commonName      = supplied
emailAddress        = optional

####################################################################
[ signing_req ]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer

basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment

8.touch index.txt

9.echo '01' > serial.txt

10.openssl ca -config openssl-ca.cnf -policy signing_policy -extensions signing_req -out servercert.pem -infiles servercert.csr

11.openssl x509 -in servercert.pem -inform PEM -out servercert.der -outform DER

12.Added servercert.der in iOS project

13.let sslsock = SSLSocketLite(inHost: "localhost", inPort: 1678)

14.Exception Domains -> +localhost

15.openssl s_server -key serverkey.pem -cert servercert.pem -accept 1678

Final version of openssl-ca.cnf:

HOME            = .
RANDFILE        = $ENV::HOME/.rnd

####################################################################
[ ca ]
default_ca  = CA_default        # The default ca section

[ CA_default ]

default_days    = 1000          # how long to certify for
default_crl_days= 30            # how long before next CRL
default_md  = sha256        # use public key default MD
preserve    = no            # keep passed DN ordering

x509_extensions = ca_extensions     # The extensions to add to the cert

email_in_dn = no            # Don't concat the email in the DN
copy_extensions = copy          # Required to copy SANs from CSR to cert

base_dir    = .
certificate = $base_dir/cacert.pem  # The CA certifcate
private_key = $base_dir/cakey.pem   # The CA private key
new_certs_dir   = $base_dir     # Location for new certs after signing
database    = $base_dir/index.txt   # Database index file
serial      = $base_dir/serial.txt  # The current serial number

unique_subject  = no            # Set to 'no' to allow creation of
                # several certificates with same subject.
####################################################################
[ req ]
default_bits        = 4096
default_keyfile     = cakey.pem
distinguished_name  = ca_distinguished_name
x509_extensions     = ca_extensions
string_mask         = utf8only

####################################################################
[ ca_distinguished_name ]
countryName         = Country Name (2 letter code)
countryName_default     = US

stateOrProvinceName     = State or Province Name (full name)
stateOrProvinceName_default = Maryland

localityName            = Locality Name (eg, city)
localityName_default        = Baltimore

organizationName            = Organization Name (eg, company)
organizationName_default    = Test CA, Limited

organizationalUnitName  = Organizational Unit (eg, division)
organizationalUnitName_default  = Server Research Department

commonName          = Common Name (e.g. server FQDN or YOUR name)
commonName_default      = localhost

emailAddress            = Email Address
emailAddress_default        = [e-mail]

####################################################################
[ ca_extensions ]

subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always, issuer
basicConstraints = critical, CA:true
keyUsage = keyCertSign, cRLSign

####################################################################
[ signing_policy ]
countryName     = optional
stateOrProvinceName = optional
localityName        = optional
organizationName    = optional
organizationalUnitName  = optional
commonName      = supplied
emailAddress        = optional

####################################################################
[ signing_req ]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer

basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment

Final version of openssl-server.cnf:

HOME            = .
RANDFILE        = $ENV::HOME/.rnd

####################################################################
[ req ]
default_bits        = 2048
default_keyfile     = serverkey.pem
distinguished_name  = server_distinguished_name
req_extensions      = server_req_extensions
string_mask         = utf8only

####################################################################
[ server_distinguished_name ]
countryName         = Country Name (2 letter code)
countryName_default     = US

stateOrProvinceName     = State or Province Name (full name)
stateOrProvinceName_default = MD

localityName            = Locality Name (eg, city)
localityName_default        = Baltimore

organizationName            = Organization Name (eg, company)
organizationName_default    = Test CA, Limited

commonName          = Common Name (e.g. server FQDN or YOUR name)
commonName_default      = localhost

emailAddress            = Email Address
emailAddress_default        = [e-mail]

####################################################################
[ server_req_extensions ]

subjectKeyIdentifier        = hash
basicConstraints        = CA:FALSE
keyUsage            = digitalSignature, keyEncipherment
subjectAltName          = @alternate_names
nsComment           = "OpenSSL Generated Certificate"

####################################################################
[ alternate_names ]

DNS.1       = localhost

iOS output:

method started
adding finished success
2016-08-30 19:17:06.619 ssltest[5418:141544] CFNetwork SSLHandshake failed (-9807)

OpenSSL s_server output(nothing happened):

Using default temp DH parameters

ACCEPT

What have I missed? And how can I solve it?

Maybe someone can share working cert with the localhost address in the commonName? Using this I will understand which exactly is the problem: cert and my pem to der converting or iOS app.