Client certificate not sent from android to ssl se

Question

Below is code I am using to create a TrustStore holding my Certificate Authority's public cert. I am creating another KeyStore to hold a "client certificate" which is one I created from a cert signing request and signed by the CA that I mentioned. For some reason, When I call sslCertSocketFactory.createSocket(), I get an exception thrown:

SSL failure: javax.net.ssl.SSLHandshakeException: javax.net.ssl.SSLProtocolException: SSL handshake terminated: ssl=0x7780ef60: Failure in SSL library, usually a protocol error
    error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure (external/openssl/ssl/s3_pkt.c:1256 0x77829d10:0x00000003)

And on the server side, I get the following :

SSL alert (write): fatal: handshake failure
2014.07.01 15:56:46 LOG3[7121:4413599744]: SSL_accept: 140890C7: error:140890C7:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:peer did not return a certificate

What am I doing wrong? How do I make sure that the socket factory pushes the cert to the server that I specify in the localKeystore?

            // Load my CA's public cert from an InputStream

            CertificateFactory cf = CertificateFactory.getInstance("X.509");
            InputStream caInput = new BufferedInputStream(cafile);
            Certificate ca = null;
            try {
                ca = cf.generateCertificate(caInput);

            } catch(Exception e) {
                Log.d(" Problem creating the CA cert: " + e.toString());
            }
            finally {
                caInput.close();
            }
            //sets this CA cert set as a KeyStore.TrustedCertificateEntry in the keystore.
            certManagerCA.trustCertificate((X509Certificate) ca);
            KeyStore keyStoreCA = certManagerCA.sslKeystore;

            tmf = TrustManagerFactory.getInstance("X509");
            tmf.init(keyStoreCA);

            localKeystore  = KeyStore.getInstance("BKS");
            try {
                //clientBKS is a bouncy castle keystore made with portecle from a public/private keypair signed by the CA I impored above
                localKeystore.load(clientBKS, "password".toCharArray());
            }
            finally {
                clientBKS.close();
            }

            // Build a KeyManager for Client auth
            kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
            kmf.init(localKeystore, keyPass.toCharArray());
           //at this point I can iterate over the localKeyStore aliases and it ONLY has my cert in it. 

            sslCertSocketFactory = (SSLCertificateSocketFactory) SSLCertificateSocketFactory.getDefault(100000);
            sslCertSocketFactory.setKeyManagers(kmf.getKeyManagers());
            sslCertSocketFactory.setTrustManagers(tmf.getTrustManagers());
            Socket test = sslCertSocketFactory.createSocket("10.22.1.100", 443);