Decrypting an OpenSSL PEM Encoded RSA private key

Question

I have an encrypted private key and I know the password.

I need to decrypt it using a Java library.

I'd prefer not to use BouncyCastle though, unless there is no other option. Based on previous experience, there is too much change and not enough documentation.

The private key is in this form:

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,56F3A98D9CFFA77A

X5h7SUDStF1tL16lRM+AfZb1UBDQ0D1YbQ6vmIlXiK....
.....
/KK5CZmIGw==
-----END RSA PRIVATE KEY-----

I believe the key data is Base64 encoded since I see \r\n after 64 characters.

I tried the following to decrypt the key:

import java.security.Key;
import java.security.KeyFactory;
import java.security.PrivateKey;
import java.security.spec.PKCS8EncodedKeySpec;
import javax.crypto.EncryptedPrivateKeyInfo;
import javax.crypto.SecretKeyFactory;
import javax.crypto.spec.PBEKeySpec;

public String decrypt(String keyDataStr, String passwordStr){
  // This key data start from "X5... to ==" 
  char [] password=passwordStr.toCharArray();
  byte [] keyDataBytes=com.sun.jersey.core.util.Base64.decode(keyDataStr);

  PBEKeySpec pbeSpec = new PBEKeySpec(password);
  EncryptedPrivateKeyInfo pkinfo = new EncryptedPrivateKeyInfo(keyDataBytes);
  SecretKeyFactory skf = SecretKeyFactory.getInstance(pkinfo.getAlgName());
  Key secret = skf.generateSecret(pbeSpec);
  PKCS8EncodedKeySpec keySpec = pkinfo.getKeySpec(secret);
  KeyFactory kf = KeyFactory.getInstance("RSA");
  PrivateKey pk=kf.generatePrivate(keySpec);
  return pk.toString();
}

I get this Exception

java.io.IOException: DerInputStream.getLength(): lengthTag=50, too big.
    at sun.security.util.DerInputStream.getLength(DerInputStream.java:561)
    at sun.security.util.DerValue.init(DerValue.java:365)
    at sun.security.util.DerValue.<init>(DerValue.java:294)
    at javax.crypto.EncryptedPrivateKeyInfo.<init> (EncryptedPrivateKeyInfo.java:84)

Am I passing the right parameter to EncryptedPrivateKeyInfo constructor?

How can I make this work?

I tried what Ericsonn suggested, with one small change since I am working Java 7, I could not use Base64.getMimeCoder() instead I used Base64.decode and I am getting this error I am getting an error like this Input length must be multiple of 8 when decrypting with padded cipher at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:750)

static RSAPrivateKey decrypt(String keyDataStr, String ivHex, String password)
            throws GeneralSecurityException, UnsupportedEncodingException
          {
            byte[] pw = password.getBytes(StandardCharsets.UTF_8);
            byte[] iv = h2b(ivHex);
            SecretKey secret = opensslKDF(pw, iv);
            Cipher cipher = Cipher.getInstance("DESede/CBC/NoPadding");
            cipher.init(Cipher.DECRYPT_MODE, secret, new IvParameterSpec(iv));
            byte [] keyBytes=Base64.decode(keyDataStr.getBytes("UTF-8"));
            byte[] pkcs1 = cipher.doFinal(keyBytes);
            /* See note for definition of "decodeRSAPrivatePKCS1" */
            RSAPrivateCrtKeySpec spec = decodeRSAPrivatePKCS1(pkcs1);
            KeyFactory rsa = KeyFactory.getInstance("RSA");
            return (RSAPrivateKey) rsa.generatePrivate(spec);
          }

          private static SecretKey opensslKDF(byte[] pw, byte[] iv)
            throws NoSuchAlgorithmException
          {
            MessageDigest md5 = MessageDigest.getInstance("MD5");
            md5.update(pw);
            md5.update(iv);
            byte[] d0 = md5.digest();
            md5.update(d0);
            md5.update(pw);
            md5.update(iv);
            byte[] d1 = md5.digest();
            byte[] key = new byte[24];
            System.arraycopy(d0, 0, key, 0, 16);
            System.arraycopy(d1, 0, key, 16, 8);
            return new SecretKeySpec(key, "DESede");
          }

          private static byte[] h2b(CharSequence s)
          {
            int len = s.length();
            byte[] b = new byte[len / 2];
            for (int src = 0, dst = 0; src < len; ++dst) {
              int hi = Character.digit(s.charAt(src++), 16);
              int lo = Character.digit(s.charAt(src++), 16);
              b[dst] = (byte) (hi << 4 | lo);
            }
            return b;
          }
          static RSAPrivateCrtKeySpec decodeRSAPrivatePKCS1(byte[] encoded)
          {
            ByteBuffer input = ByteBuffer.wrap(encoded);
            if (der(input, 0x30) != input.remaining())
              throw new IllegalArgumentException("Excess data");
            if (!BigInteger.ZERO.equals(derint(input)))
              throw new IllegalArgumentException("Unsupported version");
            BigInteger n = derint(input);
            BigInteger e = derint(input);
            BigInteger d = derint(input);
            BigInteger p = derint(input);
            BigInteger q = derint(input);
            BigInteger ep = derint(input);
            BigInteger eq = derint(input);
            BigInteger c = derint(input);
            return new RSAPrivateCrtKeySpec(n, e, d, p, q, ep, eq, c);
          }

          private static BigInteger derint(ByteBuffer input)
          {
            byte[] value = new byte[der(input, 0x02)];
            input.get(value);
            return new BigInteger(+1, value);
          }


          private static int der(ByteBuffer input, int exp)
          {
            int tag = input.get() & 0xFF;
            if (tag != exp)
              throw new IllegalArgumentException("Unexpected tag");
            int n = input.get() & 0xFF;
            if (n < 128)
              return n;
            n &= 0x7F;
            if ((n < 1) || (n > 2))
              throw new IllegalArgumentException("Invalid length");
            int len = 0;
            while (n-- > 0) {
              len <<= 8;
              len |= input.get() & 0xFF;
            }
            return len;
          }

1640 is keyDataStr.length() and 1228 is keyBytes.length

Answer 1

Java code example below shows how to construct the decryption key to obtain the underlying RSA key from an encrypted private key created using the openssl 1.0.x genrsa command; specifically from the following genrsa options that may have been leveraged:

-des encrypt the generated key with DES in cbc mode

-des3 encrypt the generated key with DES in ede cbc mode (168 bit key)

-aes128, -aes192, -aes256 encrypt PEM output with cbc aes

Above options result in encrypted RSA private key of the form ...

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AAA,BBB
...

Where AAA would be one of:

DES-CBC, DES-EDE3-CBC, AES-128-CBC, AES-192-CBC, AES-256-CBC

AND BBB is the hex-encoded IV value

KeyFactory factory = KeyFactory.getInstance("RSA");
KeySpec keySpec = null;
RSAPrivateKey privateKey = null;

Matcher matcher = OPENSSL_ENCRYPTED_RSA_PRIVATEKEY_PATTERN.matcher(pemContents);
if (matcher.matches())
{
    String encryptionDetails = matcher.group(1).trim(); // e.g. AES-256-CBC,XXXXXXX
    String encryptedKey = matcher.group(2).replaceAll("\\s", ""); // remove tabs / spaces / newlines / carriage return etc

    System.out.println("PEM appears to be OpenSSL Encrypted RSA Private Key; Encryption details : "
        + encryptionDetails + "; Key : " + encryptedKey);

    byte[] encryptedBinaryKey = java.util.Base64.getDecoder().decode(encryptedKey);

    String[] encryptionDetailsParts = encryptionDetails.split(",");
    if (encryptionDetailsParts.length == 2)
    {
        String encryptionAlgorithm = encryptionDetailsParts[0];
        String encryptedAlgorithmParams = encryptionDetailsParts[1]; // i.e. the initialization vector in hex

        byte[] pw = new String(password).getBytes(StandardCharsets.UTF_8);
        byte[] iv = fromHex(encryptedAlgorithmParams);

        MessageDigest digest = MessageDigest.getInstance("MD5");

        // we need to come up with the encryption key

        // first round digest based on password and first 8-bytes of IV ..
        digest.update(pw);
        digest.update(iv, 0, 8);

        byte[] round1Digest = digest.digest(); // The digest is reset after this call is made.

        // second round digest based on first round digest, password, and first 8-bytes of IV ...
        digest.update(round1Digest);
        digest.update(pw);
        digest.update(iv, 0, 8);

        byte[] round2Digest = digest.digest();

        Cipher cipher = null;
        SecretKey secretKey = null;
        byte[] key = null;
        byte[] pkcs1 = null;

        if ("AES-256-CBC".equals(encryptionAlgorithm))
        {
            cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");

            key = new byte[32]; // 256 bit key  (block size still 128-bit)
            System.arraycopy(round1Digest, 0, key, 0, 16);
            System.arraycopy(round2Digest, 0, key, 16, 16);

            secretKey = new SecretKeySpec(key, "AES");
        }
        else if ("AES-192-CBC".equals(encryptionAlgorithm))
        {
            cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");

            key = new byte[24]; // key size of 24 bytes
            System.arraycopy(round1Digest, 0, key, 0, 16);
            System.arraycopy(round2Digest, 0, key, 16, 8);

            secretKey = new SecretKeySpec(key, "AES");
        }
        else if ("AES-128-CBC".equals(encryptionAlgorithm))
        {
            cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");

            key = new byte[16]; // 128 bit key
            System.arraycopy(round1Digest, 0, key, 0, 16);

            secretKey = new SecretKeySpec(key, "AES");
        }
        else if ("DES-EDE3-CBC".equals(encryptionAlgorithm))
        {
            cipher = Cipher.getInstance("DESede/CBC/PKCS5Padding");

            key = new byte[24]; // key size of 24 bytes
            System.arraycopy(round1Digest, 0, key, 0, 16);
            System.arraycopy(round2Digest, 0, key, 16, 8);

            secretKey = new SecretKeySpec(key, "DESede");
        }
        else if ("DES-CBC".equals(encryptionAlgorithm))
        {
            cipher = Cipher.getInstance("DES/CBC/PKCS5Padding");

            key = new byte[8]; // key size of 8 bytes
            System.arraycopy(round1Digest, 0, key, 0, 8);

            secretKey = new SecretKeySpec(key, "DES");
        }

        cipher.init(Cipher.DECRYPT_MODE, secretKey, new IvParameterSpec(iv));

        pkcs1 = cipher.doFinal(encryptedBinaryKey);

        keySpec = pkcs1ParsePrivateKey(pkcs1);

        privateKey = (RSAPrivateKey) factory.generatePrivate(keySpec);
    }
}

The regular expression ...

static final String OPENSSL_ENCRYPTED_RSA_PRIVATEKEY_REGEX = "\\s*" 
+ "-----BEGIN RSA PUBLIC KEY-----" + "\\s*"
+ "Proc-Type: 4,ENCRYPTED" + "\\s*"
+ "DEK-Info:" + "\\s*([^\\s]+)" + "\\s+"
+ "([\\s\\S]*)"
+ "-----END RSA PUBLIC KEY-----" + "\\s*";

static final Pattern OPENSSL_ENCRYPTED_RSA_PRIVATEKEY_PATTERN = Pattern.compile(OPENSSL_ENCRYPTED_RSA_PRIVATEKEY_REGEX);

the fromHex(...) method ...

public static byte[] fromHex(String hexString)
{
    byte[] bytes = new byte[hexString.length() / 2];
    for (int i = 0; i < hexString.length(); i += 2)
    {
        bytes[i / 2] = (byte) ((Character.digit(hexString.charAt(i), 16) << 4)
            + Character.digit(hexString.charAt(i + 1), 16));
    }
    return bytes;
}

Answer 2

You need to use a non-standard, OpenSSL method for deriving the decryption key. Then use that to decrypt the PKCS-#1–encoded key—what you are working with is not a PKCS #8 envelope. You'll also need the IV from the header as input to these processes.

It looks something like this:

  static RSAPrivateKey decrypt(String keyDataStr, String ivHex, String password)
    throws GeneralSecurityException
  {
    byte[] pw = password.getBytes(StandardCharsets.UTF_8);
    byte[] iv = h2b(ivHex);
    SecretKey secret = opensslKDF(pw, iv);
    Cipher cipher = Cipher.getInstance("DESede/CBC/PKCS5Padding");
    cipher.init(Cipher.DECRYPT_MODE, secret, new IvParameterSpec(iv));
    byte[] pkcs1 = cipher.doFinal(Base64.getMimeDecoder().decode(keyDataStr));
    /* See note for definition of "decodeRSAPrivatePKCS1" */
    RSAPrivateCrtKeySpec spec = decodeRSAPrivatePKCS1(pkcs1);
    KeyFactory rsa = KeyFactory.getInstance("RSA");
    return (RSAPrivateKey) rsa.generatePrivate(spec);
  }

  private static SecretKey opensslKDF(byte[] pw, byte[] iv)
    throws NoSuchAlgorithmException
  {
    MessageDigest md5 = MessageDigest.getInstance("MD5");
    md5.update(pw);
    md5.update(iv);
    byte[] d0 = md5.digest();
    md5.update(d0);
    md5.update(pw);
    md5.update(iv);
    byte[] d1 = md5.digest();
    byte[] key = new byte[24];
    System.arraycopy(d0, 0, key, 0, 16);
    System.arraycopy(d1, 0, key, 16, 8);
    return new SecretKeySpec(key, "DESede");
  }

  private static byte[] h2b(CharSequence s)
  {
    int len = s.length();
    byte[] b = new byte[len / 2];
    for (int src = 0, dst = 0; src < len; ++dst) {
      int hi = Character.digit(s.charAt(src++), 16);
      int lo = Character.digit(s.charAt(src++), 16);
      b[dst] = (byte) (hi << 4 | lo);
    }
    return b;
  }

This is already a lot of code, so I will link to another answer for the definition of the decodeRSAPrivatePKCS1() method.