TLS 1.2 ECDHE_RSA signature

Question

I'm currently working on a Java TLS server. I'm trying to get the following CipherSuite to work : TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

When I test it using openssl s_client I get the following error after the ServerKeyExchange message :

140735242416208:error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type:t1_lib.c:1130:

Here is the TLS message as seen in Wireshark

The Handshake fails on a decode_error fatal error.

So I guess the client doesn't like the signature algorithm chosen.

But I am only using the default SignatureAndHashAlgorithm for now as per RFC 5246 Section-7.4.1.4.1

If the negotiated key exchange algorithm is one of (RSA, DHE_RSA, DH_RSA, RSA_PSK, ECDH_RSA, ECDHE_RSA), behave as if client had sent the value {sha1,rsa}.

(I'm still checking if the client do offer theses default values though)

Since I'm doing ECDHE_RSA I believe I should hash and sign the serverECDHparams as per RFC 4492 Section 5.4 (First post here so only 2 links sorry :) )

ServerKeyExchange.signed_params.sha_hash
        SHA(ClientHello.random + ServerHello.random +
                                          ServerKeyExchange.params);
struct {
    select (KeyExchangeAlgorithm) {
        case ec_diffie_hellman:
            ServerECDHParams params;
            Signature signed_params;
    };
} ServerKeyExchange;

And I should do this as per RFC 2246 Section 7.4.3

select (SignatureAlgorithm) {   
    case rsa:
        digitally-signed struct {
            opaque md5_hash[16];
            opaque sha_hash[20];
        };
} Signature;


md5_hash
MD5(ClientHello.random + ServerHello.random + ServerParams);

sha_hash
SHA(ClientHello.random + ServerHello.random + ServerParams);

My Java code regarding signing the serverParams :

private byte[] getSignedParams(ChannelBuffer params)
        throws NoSuchAlgorithmException, DigestException, 
        SignatureException, InvalidKeyException {
    byte[] signedParams = null;
    ChannelBuffer signAlg = ChannelBuffers.buffer(2);
    MessageDigest md5 = MessageDigest.getInstance("MD5");
    MessageDigest sha = MessageDigest.getInstance("SHA-1");
    switch (session.cipherSuite.sign) {
        case rsa:
            signAlg.writeByte(2); // 2 for SHA1
            sha.update(clientRandom);
            sha.update(serverRandom);
            sha.update(params.toByteBuffer());
            md5.update(clientRandom);
            md5.update(serverRandom);
            md5.update(params.toByteBuffer());
            signedParams = concat(md5.digest(), sha.digest());
        break;
    }
    signAlg.writeByte(session.cipherSuite.sign.value); // for RSA he byte is one
    ChannelBuffer signLength = ChannelBuffers.buffer(2);
    signLength.writeShort(signedParams.length);
    return concat(signAlg.array(),concat(signLength.array(),signedParams));
}

So my question is basically : Am I wrong about all this ? and if so, what am I doing wrong ?

Thank you for your time ! :)

Answer 1

It's me again, I seem to have fixed my particular problem 2 things I noted :

1. Regarding my Java code, the MessageDigest class only does hashing no signing so I now use the Signature class instead.
2. It seems I only need to sign using SHA1 in TLS1.2 I don't need to do MD5 at all.

The second item is what I should have found in the RFC but didn't (maybe it is written somewhere, I don't know) I think this could be useful for people even if they're not doing Java ;)

How my code looks now :

private byte[] getSignedParams(ChannelBuffer params)
        throws NoSuchAlgorithmException, DigestException, 
        SignatureException, InvalidKeyException {
    byte[] signedParams = null;
    Signature signature = Signature.getInstance(selectedSignAndHash.toString());
    ChannelBuffer signAlg = ChannelBuffers.buffer(2);
    signAlg.writeByte(selectedSignAndHash.hash.value);
    signature.initSign(privateKey);

    signature.update(clientRandom);
    signature.update(serverRandom);
    signature.update(params.toByteBuffer());

    signedParams = signature.sign();

    signAlg.writeByte(session.cipherSuite.sign.value);
    ChannelBuffer signLength = ChannelBuffers.buffer(2);
    signLength.writeShort(signedParams.length);
    return concat(signAlg.array(), concat(signLength.array(), signedParams));
}

The code is different because in between I added a function to choose the SignatureAndHashAlgorithm to use from the list the client provides. But you could modify this to only respond using SHA1withRSA as this seems to be the default HashAndSignatureAlgorithm.