I'm currently working on a Java TLS server. I'm trying to get the following CipherSuite to work : TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
When I test it using openssl s_client I get the following error after the ServerKeyExchange message :
140735242416208:error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type:t1_lib.c:1130:
Here is the TLS message as seen in Wireshark
The Handshake fails on a decode_error fatal error.
So I guess the client doesn't like the signature algorithm chosen.
But I am only using the default SignatureAndHashAlgorithm for now as per RFC 5246 Section-7.4.1.4.1
If the negotiated key exchange algorithm is one of (RSA, DHE_RSA, DH_RSA, RSA_PSK, ECDH_RSA, ECDHE_RSA), behave as if client had sent the value {sha1,rsa}.
(I'm still checking if the client do offer theses default values though)
Since I'm doing ECDHE_RSA I believe I should hash and sign the serverECDHparams as per RFC 4492 Section 5.4 (First post here so only 2 links sorry :) )
ServerKeyExchange.signed_params.sha_hash
SHA(ClientHello.random + ServerHello.random +
ServerKeyExchange.params);
struct {
select (KeyExchangeAlgorithm) {
case ec_diffie_hellman:
ServerECDHParams params;
Signature signed_params;
};
} ServerKeyExchange;
And I should do this as per RFC 2246 Section 7.4.3
select (SignatureAlgorithm) {
case rsa:
digitally-signed struct {
opaque md5_hash[16];
opaque sha_hash[20];
};
} Signature;
md5_hash
MD5(ClientHello.random + ServerHello.random + ServerParams);
sha_hash
SHA(ClientHello.random + ServerHello.random + ServerParams);
My Java code regarding signing the serverParams :
private byte[] getSignedParams(ChannelBuffer params)
throws NoSuchAlgorithmException, DigestException,
SignatureException, InvalidKeyException {
byte[] signedParams = null;
ChannelBuffer signAlg = ChannelBuffers.buffer(2);
MessageDigest md5 = MessageDigest.getInstance("MD5");
MessageDigest sha = MessageDigest.getInstance("SHA-1");
switch (session.cipherSuite.sign) {
case rsa:
signAlg.writeByte(2); // 2 for SHA1
sha.update(clientRandom);
sha.update(serverRandom);
sha.update(params.toByteBuffer());
md5.update(clientRandom);
md5.update(serverRandom);
md5.update(params.toByteBuffer());
signedParams = concat(md5.digest(), sha.digest());
break;
}
signAlg.writeByte(session.cipherSuite.sign.value); // for RSA he byte is one
ChannelBuffer signLength = ChannelBuffers.buffer(2);
signLength.writeShort(signedParams.length);
return concat(signAlg.array(),concat(signLength.array(),signedParams));
}
So my question is basically : Am I wrong about all this ? and if so, what am I doing wrong ?
Thank you for your time ! :)