breaking down RSA/ECB/OAEPWITHSHA-256ANDMGF1PADDIN

So Java has a mode called RSA/ECB/OAEPWITHSHA-256ANDMGF1PADDING. What does that even mean?

RFC3447, Public-Key Cryptography Standards (PKCS) #1: RSA Cryptography Specifications Version 2.1, section 7.1.2 Decryption operation says Hash and MGF are both options for RSAES-OAEP-DECRYPT. MGF is it's own function, defined in Section B.2.1 MGF1 and that has it's own Hash "option" as well.

Maybe the Hash "option" in RSAES-OAEP-DECRYPT and MGF1 are supposed to be the same or maybe they're not, it is unclear to me. If they are then I guess when you have RSA/ECB/OAEPWITHSHA-256ANDMGF1PADDING that means sha256 should be used for both. But if they're not supposed to be the same then you could have sha256 used for RSAES-OAEP-DECRYPT and, for example, sha1 used for MGF1. And if that's the case then what function is sha256 supposed to be used for? And what hash algorithm is supposed to be used for the other function?

And what does ECB mean in this context? ECB is a symmetric block cipher mode. Electronic Code Book. Maybe it's supposed to mean how Java deals with plaintext's that are larger than the modulo? Like maybe splits the plaintext into chunks that are as big as the modulo and then encrypts each one with RSA and concatenates them together? I'm just guessing..

The default for OAEP is to use SHA-1 for MGF1. Note that the hash chosen doesn't have that much impact on the security of OAEP, so mostly it will be left to this default.

We can easily test this by testing it against "OAEPPadding" and OAEPParameterSpec:

// --- we need a key pair to test encryption/decryption
KeyPairGenerator kpg = KeyPairGenerator.getInstance("RSA");
kpg.initialize(1024); // speedy generation, but not secure anymore
KeyPair kp = kpg.generateKeyPair();
RSAPublicKey pubkey = (RSAPublicKey) kp.getPublic();
RSAPrivateKey privkey = (RSAPrivateKey) kp.getPrivate();

// --- encrypt given algorithm string
Cipher oaepFromAlgo = Cipher.getInstance("RSA/ECB/OAEPWITHSHA-256ANDMGF1PADDING");
oaepFromAlgo.init(Cipher.ENCRYPT_MODE, pubkey);
byte[] ct = oaepFromAlgo.doFinal("owlstead".getBytes(StandardCharsets.UTF_8));

// --- decrypt given OAEPParameterSpec
Cipher oaepFromInit = Cipher.getInstance("RSA/ECB/OAEPPadding");
OAEPParameterSpec oaepParams = new OAEPParameterSpec("SHA-256", "MGF1", new MGF1ParameterSpec("SHA-1"), PSpecified.DEFAULT);
oaepFromInit.init(Cipher.DECRYPT_MODE, privkey, oaepParams);
byte[] pt = oaepFromInit.doFinal(ct);
System.out.println(new String(pt, StandardCharsets.UTF_8));

The code will fail with a padding related exception if you substitute "SHA-256" for the MGF1 as parameter.

The reason why the extended algorithm is needed at all is compatibility with other Cipher algorithms. Code written for e.g. "RSA/ECB/PKCS1Padding" doesn't use any parameters, let alone OAEP parameters. So without the longer string OAEP cannot function as drop in replacement.

The mode of operation "ECB" doesn't mean anything in this context, it should have been "None" or it should have been left out completely. You can only encrypt a single block using the RSA implementation of the SunRSA provider.

If you want to encrypt more data, create a random (AES) symmetric key and encrypt that using OAEP. Then use the AES key to encrypt your specific data. This is called a hybrid cryptosystem as it uses both asymmetric and symmetric primitives to encrypt data.