Generate Subject Hash of X509Certificate in Java

I'm currently trying to generate the subject hash by using the Java Security API and BouncyCastle.

Here's what I do, when I use the Openssl Library:

openssl x509 -in  /Users/Sn0wfreezeDev/Downloads/Test.pem -hash

This generates a short 8 digit hash 1817886a

This is my Java code

X509Certificate cert = CertManager.getCertificate(number, c);  
MessageDigest sha1 = MessageDigest.getInstance("SHA1");
System.out.println("  Subject " + cert.getSubjectDN());
System.out.println("   Issuer  " + cert.getIssuerDN());
sha1.update(cert.getSubjectDN().getName().getBytes());
String hexString =  bytesToHex(sha1.digest());
System.out.println("   sha1    " + hexString);
System.out.println();

This generates a short 8 digit hash 1817886a

There are two forms of this from OpenSSL:

$ cd openssl-1.0.2-src
$ grep -R X509_subject_name_hash *
...
crypto/x509/x509.h:unsigned long X509_subject_name_hash(X509 *x);
crypto/x509/x509.h:unsigned long X509_subject_name_hash_old(X509 *x);
crypto/x509/x509_cmp.c:unsigned long X509_subject_name_hash(X509 *x)
crypto/x509/x509_cmp.c:unsigned long X509_subject_name_hash_old(X509 *x)
...

Generate Subject Hash of X509Certificate in Java...

Here is the source for them from crypto/x509/x509_cmp.c:

unsigned long X509_subject_name_hash(X509 *x)
{
    return (X509_NAME_hash(x->cert_info->subject));
}

#ifndef OPENSSL_NO_MD5
unsigned long X509_subject_name_hash_old(X509 *x)
{
    return (X509_NAME_hash_old(x->cert_info->subject));
}
#endif

And finally:

unsigned long X509_NAME_hash(X509_NAME *x)
{
    unsigned long ret = 0;
    unsigned char md[SHA_DIGEST_LENGTH];

    /* Make sure X509_NAME structure contains valid cached encoding */
    i2d_X509_NAME(x, NULL);
    if (!EVP_Digest(x->canon_enc, x->canon_enclen, md, NULL, EVP_sha1(),
                    NULL))
        return 0;

    ret = (((unsigned long)md[0]) | ((unsigned long)md[1] << 8L) |
           ((unsigned long)md[2] << 16L) | ((unsigned long)md[3] << 24L)
        ) & 0xffffffffL;
    return (ret);
}

#ifndef OPENSSL_NO_MD5
unsigned long X509_NAME_hash_old(X509_NAME *x)
{
    EVP_MD_CTX md_ctx;
    unsigned long ret = 0;
    unsigned char md[16];

    /* Make sure X509_NAME structure contains valid cached encoding */
    i2d_X509_NAME(x, NULL);
    EVP_MD_CTX_init(&md_ctx);
    EVP_MD_CTX_set_flags(&md_ctx, EVP_MD_CTX_FLAG_NON_FIPS_ALLOW);
    if (EVP_DigestInit_ex(&md_ctx, EVP_md5(), NULL)
        && EVP_DigestUpdate(&md_ctx, x->bytes->data, x->bytes->length)
        && EVP_DigestFinal_ex(&md_ctx, md, NULL))
        ret = (((unsigned long)md[0]) | ((unsigned long)md[1] << 8L) |
               ((unsigned long)md[2] << 16L) | ((unsigned long)md[3] << 24L)
            ) & 0xffffffffL;
    EVP_MD_CTX_cleanup(&md_ctx);

    return (ret);
}
#endif

i2d_X509_NAME encodes a X509_NAME into a standard representation using RFC 2459 (and elsewhere). Its used, for example, in certificate subject and issuer names.

You can see what OpenSSL uses for the name string with commands like openssl x509 -in <cert> -text -noout. It will look similar to C=US, ST=California, L=Mountain View, O=Google Inc, CN=www.google.com (taken from a Google certificate).

Generate Subject Hash of X509Certificate in Java...

In the big picture, you are generating a hash of the Subject's Distinguished Name string and returning an unsigned long. The unsigned long is effectively a truncated hash.

X509_subject_name_hash uses SHA-1, and X509_subject_name_hash_old uses MD5.