TLS-secured TCP server and client with self-signed

I'm working on an app that connects two Android devices via Wifi so they can exchange files/data using TCP. Since Android Oreo (API level 26) there's finally an official API for this: WifiManager.startLocalOnlyHotspot(). This creates a Wifi hotspot/network without internet access. The documentation says:

Applications should also be aware that this network will be shared with other applications. Applications are responsible for protecting their data on this network (e.g., TLS).

I have no experience in using TLS when it comes to connecting two devices via TCP, so I searched around and found some approaches mentioning self-signed certificates. I'm not sure wether this is a good practice; I can't get it working anyway. Any help is appreciated!

What I did so far:

I created a self-signed certificate using OpenSSL as described in this answer:
```
openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 10
```

I created a new keystore with the latest (March 2018) Bouncy Castle provider .jar file and added cert.pem to it. The following snippet is derived from this great blog article, more specifically from the sample app it features.

ALIAS=`openssl x509 -inform PEM -subject_hash -noout -in cert.pem`

keytool -import -v -trustcacerts \
        -alias $ALIAS \
        -file cert.pem \
        -keystore keystore_output_file \
        -storetype BKS \
        -providerclass org.bouncycastle.jce.provider.BouncyCastleProvider \
        -providerpath bcprov-jdk15on-159.jar \
        -storepass my_keystore_password

I've added keystore_output_file to the src/res/raw/ folder of my app and initialized a SSLContext. Then my app creates a SSLServerSocket when acting as server or a SSLSocket when acting as client. Originally I found this approach here.

// Exception handling omitted
KeyStore keyStore = KeyStore.getInstance("BKS");
InputStream certStore = context.getResources().openRawResource(R.raw.keystore_output_file);

keyStore.load(certStore, "my_keystore_password".toCharArray());

TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
trustManagerFactory.init(keyStore);

KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
keyManagerFactory.init(keyStore, "my_key_password".toCharArray());

SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(keyManagerFactory.getKeyManagers(), trustManagerFactory.getTrustManagers(), new SecureRandom());

...continuing as server:

SSLServerSocketFactory sslServerSocketFactory = sslContext.getServerSocketFactory();
SSLServerSocket sslServerSocket = (SSLServerSocket) sslServerSocketFactory.createServerSocket(port);

SSLSocket sslClientSocket = (SSLSocket) sslServerSocket.accept();

...continuing as client:

SSLSocketFactory sslSocketFactory = sslContext.getSocketFactory();
SSLSocket sslSocket = (SSLSocket) sslSocketFactory.createSocket(ipAddress, port);

sslSocket.startHandshake();

The Problem:

The connection fails. There are different error messages depending on what version of Android the server device is running, but both are mentioning problems with ciphers. The Android versions of my testing devices are 4.3 and 7.1.1. The stacktraces are:

Server: Android 7.1.1

javax.net.ssl.SSLHandshakeException: Handshake failed                                                                                 
    at com.android.org.conscrypt.OpenSSLSocketImpl.startHandshake(OpenSSLSocketImpl.java:429)                                         
    at com.android.org.conscrypt.OpenSSLSocketImpl.waitForHandshake(OpenSSLSocketImpl.java:682)                                       
    at com.android.org.conscrypt.OpenSSLSocketImpl.getInputStream(OpenSSLSocketImpl.java:644)                                         
    at com.candor.tlstcptest.ServerWorkerThread.run(ServerWorkerThread.java:46)                                  
Caused by: javax.net.ssl.SSLProtocolException: SSL handshake aborted: ssl=0x8b0f8a40: Failure in SSL library, usually a protocol error
error:100000b8:SSL routines:OPENSSL_internal:NO_SHARED_CIPHER (external/boringssl/src/ssl/s3_srvr.c:1059 0x99b4286a:0x00000000)       
    at com.android.org.conscrypt.NativeCrypto.SSL_do_handshake(Native Method)                                                         
    at com.android.org.conscrypt.OpenSSLSocketImpl.startHandshake(OpenSSLSocketImpl.java:357)

Server: Android 4.3

javax.net.ssl.SSLException: Could not find any key store entries to support the enabled cipher suites.
    at org.apache.harmony.xnet.provider.jsse.OpenSSLServerSocketImpl.checkEnabledCipherSuites(OpenSSLServerSocketImpl.java:232)
    at org.apache.harmony.xnet.provider.jsse.OpenSSLServerSocketImpl.accept(OpenSSLServerSocketImpl.java:177)
    at com.candor.tlstcptest.ServerThread.run(ServerThread.java:71)

Now I'm stuck, I don't even know how to start to resolve this problem and I haven't yet found helpful information online. I wonder if there is really no offical documentation on this topic... As I said, any help is appreciated! Thanks

The problem here is that you've created a keystore that only contains the certificate, not its private key. (This is what keytool -import ... does.)

One way to create a keystore that has a private key entry (with its corresponding certificate) would be to create a PKCS#12 store from OpenSSL and then convert it into BKS via keytool (more or less the same principle as here).

openssl pkcs12 -export -in cert.pem -inkey key.pem -out store.p12

Then, convert it into BKS using keytool -importkeystore (not just -import). I haven't tried the exact command, but this should be something like this:

keytool -importkeystore \
        -srckeystore store.p12 -srcstoretype PKCS12 \
        -destkeystore store.jks -deststoretype BKS \
        -providerclass org.bouncycastle.jce.provider.BouncyCastleProvider \
        -providerpath bcprov-jdk15on-159.jar

(Check the keytool documentation for the exact options you may also need.)

This should result in a store.jks keystore that also contains the private key. That keystore should only be used on the server side, as the keystore, not as the truststore on the client side." (The one you already had can be used as a truststore, on the client side.)

A couple of side notes:

Another thing to check with SSL/TLS is the identity in the certificate (not just trusting that the certificate is genuine and issued by a party you know). This is not always checked by default in Java (depending on the options you use).
For this particular type of application (essentially ad-hoc connections), you may want to generate the certificate/private key pair on the server upon installation, show its fingerprint to the user, then have a looser trust manager on the client that displays the fingerprint it has to validate it interactively (and possibly remember it).

If you do indeed manage to validate an individual certificate for the remote device explicitly, you might be able to do it securely without checking the name in the certificate (as long as the client can check it's connecting to a device with that exact certificate).

I'm not familiar enough with startLocalOnlyHotspot, but I suspect many of those details will depend on the IP address used by the hotspot. I'd imagine it also embeds some form of DHCP server to give the client an IP address, but I'm not sure how the client can get the IP address of the hotspot device itself (there might be some mDNS integration, for example).