RSA Encryption using openssl, variable length for

Question

I am trying to encrypt a string "HELLO" using RSA encryption with PKCS1 Padding. I am using openssl for generating the keypair and perform encryption.

Problem: I am getting the generated encypted text with variable length.

Private Key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBbzdOdmUwZlg3L2NjVExibWhQOWNYUVI5WWFrMi9Oa3Q2bVkwNkpjNEJjWmswemdiCnZ4ek5mTC9sNVQzd1pQcDBWa204RGozcjNVUStMMXlyRjF6bGpBSXE5MlJQcnVWRFpacTJOcXJFelhyTUZxcUMKaWhEMU04dW1EVTNNUTNCMnYxdzdsVU1uWE1Ec21LMDVxbWJRQnRzckp1ck93Mm95a2VrT0dRdndEQWRjZmIydQpVcW5NNVc2OUp3eXNQaTVkc3VhOFBXM0FIK3o0b1lRZXhrOFh4KzRWSllTNkxvVGpaeGFPNysvNXRHMVJhUUc1CklHbFQrZUFjTmtXK1pTKzlzWldXdWRGVldNeGVQaE9jSVdmL1lsNW9HSm5LaUVKRU12bkdzRmpKRlpicTVCajgKaCtyb0xjdWZvWnNPYnVRM3draFpIakNKakFQRmlhcktIR3VjQXdJREFRQUJBb0lCQUVObGp0SnN3OFN5TWw2TApOdUV1MlhJYWRSeU9OR3RwWnZRSVhHUWVUaHhpa0FRK0ozNUpkM2lTOFMydWtKaWlYTjlod0NFRThXVEp3ZWh4CitLYkh3bUlnU2x4TVdsb2hYZHdwRkRmeG10RVdhNHd4YTBrNHJodEREdnlQT1RmU3g3RUZZaXFUMlZMZ3d3Z2cKMllzRWhVVXRXdGpNOTQrUTIzMGpQSUNuWDQ4emwvZnk2YStldzZFbmNpRzNpcEppczdPc3ZxVmc4OGluSDNPWgp3Z3R4ZmpHa2RIL0NzUGk3eVYzMHdmMTZWNThFcmtlamxtOHMrRGtvcTNpRXhBekMvL3Q2TnY0Vlh5VEJYNjRHClV3RjVHdDcyL0pLTldMdytpU1dKdWtkR21wcCtxQlBROHc3QU0xeHUyalc4SFZKbUhUZGRLazBSby9KbVNwNWgKelhic0w4RUNnWUVBMVFEcVFaU0x0SElsVU1ibC9SOWZLWWJVV3JhbE9wTDdjM0EwVnplUWlya2tEV1VnSE9MTgp5bk9NYlUxcitCUFRiN1FoMytsWWlod2tHVHZsaWR6cC9BVDdDSHlpYWVnZVh2RWY1YVFNVGlhajU2UCtkNFh4CmtnZ3ZOZmI5R0VEMkovdi82SS91cTlZb1N6aUY0R0M3R09rYWxtamNSNmJ2U2trdC9QRitjWk1DZ1lFQXhMN0cKMk8vQTVQaGoraHpGbFVhNDFLb1F6WTJ4dUkrTnNFZUZSbStsejJ4M00weHIrRlBGelhnQjFIOHBNcGNxMTlFMAp6VGNiODYvck40eFZmMkJwNnNCOUFaSG42NWM3cy9rZUllNTRnMVNsRlN2NEJ1SDgyVGZnUzAwKzUrZ0hMN0I0CmY0R2ZXVm1jQkp6Sng1Njl2Ynh5TUpnSUJkRlMranIySzhFVmNkRUNnWUJsZlF3Z3R5QmhsOG1EMVdFYjIrakIKcG9aanVvYTdnL0RmTGVVaWNVMGphOWRyc25VOGppd3pmQjY2clpyMkZxdzFtd1JRNWFQVUdsaHcxWU1tenVXRgp1UEhMTjJkRFF0L0ZxT25qeHVnK0RGY29OWkUzYmRDT0hld3dzRUFqWUVCcENMOEVGaUJJSzhBREp5bFNvWVJECitEMDgwVFd6eGl4Wm5zaVVycFBxbFFLQmdRQ21EdXZnNElPci91YkxqUXRMa25TZ3NxWWFDTFZ2TkdMLzNXM2sKNUlsTHlKM1hnM0pDRjlzRk1IeEM4YzFBOFF4dE9odlo5ZUMyenMyMmhkRUlBTVdGS1ZDWlNhQTVYZDA3YTFGKwpqRHZWaFZVTXlKN0FtRytFbTcvSmJ2cGdYSnZzS1lPYkF6b08vUzVOU2xTWkV5Zk5CRWhSTVpEYjhEQ2NHYlg3CmVpM3ljUUtCZ0dJcDF2VWpRYVFaM3hhUGJ3eGpKcHlkSWQ1bk0vTllDb2doMTl6Z0tIZlNVOGxRenFsWHB0ZEEKenloOEYyLzQ5cWNuVEVnZlh0bmJ4bTRKbTV5ck5xdHUyVDFpdWt5NStST2Vud1Y1VjFqRVZETDgrZ0VsOGt2cApvVUlNSFg3ZThWYW5FTGptSWo4WU1DcGtEOFQwOVFGajhzSU5DaVZNMjk3QUs4REwyWjhFCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==

Public Key: LS0tLS1CRUdJTiBSU0EgUFVCTElDIEtFWS0tLS0tTUlJQkNnS0NBUUVBNFdXSE5obm1nUkVISVk3aVlLamVraDBmUEtxbGVlUnVVNjdnanljbllLYlJKaEpaeXF4c09BUmpubWZaaGI1ZXg5K3gxTzVBWXNPVmo5MEowanRBZk82MXhMamhHZ2F1b3NnKy9KbUhUaFp3U2tQc25HalhCLy9HNHNoWU92bUY3QndITjFNK0RUQmFGcm96NXh0RU5nMDB5YzM4cmtFYlJzNWpJSXUrTkxZRXZCZVd0eXpkK1ZqWjBuUm1nekJ5bkMxQ1VZL0lvc2lLUEZEbHRhMkhBdlFZaGxFazBGalBReW1pbEMwT1hKNXZqTVdJU2hYU2NMWklmOEhhekZySVNlUjBFNzkyb0xJL04ydzdrRkd6SmdjRFUxbFMxcWRJUFRGcTR5NFVHejB0NnExbWFxNmdOMGRIVUc2cVZWQmVEUUdSaDdpMUM2aEFLakFiV2t4Vm9RSURBUUFCLS0tLS1FTkQgUlNBIFBVQkxJQyBLRVktLS0tLQ==

Encrypted Text for HELLO in Run1: XuKAosKxw6BWXcK/w7cgMBww4oCZxLHDgO+sggXigJQR4oC5w7o3SAFbTTHiiJrDsuKJoM6p4oC5YMOf4oCwC8ODYsOa76O/MQ5BAUF2w5xMw7vDrkzigJPDq30Bw4bCoRnDgWrLnHHDukNvdcOuKkLCtsOAw6rigJnDseKAucO6G1EcwqfFkzPDiyLDnAjigKHDmUbDoxHCu8O8WU9LwrXigKDDssOvw7zDrhNfw7gCw7bvo78Jw4hp4oiGc8OrxbhTwrfiiI9mw5xQGgg6wrco76yCw6V1fg7DlDbDknPLnWY7KuKAmuKAnU3DqsOcwqzEsRg7y5g7O3RmdMOcD8OZw4TiiaXDoU7LmcWSw6fDmeKAucWTGBDFkiTiiYjiiJ5JXA3DksOuw4ldd8uH76yCUQwm4oKsw7nDhe+sgR0RUA/iiJ4vH2gTO+KAoW7DlcK/w7o8y5tFZHrCsGfigJrDocKwy5zDjMK7w4pKCibDguKInnkbUuKImsuYy5rigKDCrsO/C1HDo3LCrMOqwrXigJzigJ7DqeKAmuKIj3LDjgnDscOlwrZbDzVZw5PDmXlYO8ORFA==

Encrypted Text for HELLO in Run2: E0dedDrigJxBw5jCqhwLRA4yHcOzwqgVa1HCrmkWKcua4omlD+KEoi85G1csw7k=


NOTE: All data above is Base64Encoded.

I am using the below code to generate KeyPair and extract Private and Public Keys:

RSA *rsaKeyPair = NULL;
EVP_PKEY *PrivateKey = NULL;
rsaKeyPair = RSA_new();

BIGNUM *e = NULL;
e = BN_new();
BN_set_word(e, 65537);

//Generating KeyPair
RSA_generate_key_ex(rsaKeyPair, 2048, e, NULL);

PrivateKey = EVP_PKEY_new();

BIO *pri = BIO_new(BIO_s_mem());
BIO *pub = BIO_new(BIO_s_mem());

PEM_write_bio_RSAPrivateKey(pri, rsaKeyPair, NULL, NULL, 0, NULL, NULL);
PEM_write_bio_RSAPublicKey(pub, rsaKeyPair);

size_t pri_len = BIO_pending(pri);
size_t pub_len = BIO_pending(pub);

char *pri_key = malloc(pri_len + 1);
char *pub_key = malloc(pub_len + 1);

BIO_read(pri, pri_key, pri_len);
BIO_read(pub, pub_key, pub_len);

pri_key[pri_len] = '\0';
pub_key[pub_len] = '\0';

NSString *priK = [[[NSString stringWithFormat:@"%s",pri_key] dataUsingEncoding:NSUTF8StringEncoding] base64EncodedString];
NSString *privateKey = [[priK componentsSeparatedByCharactersInSet:[NSCharacterSet newlineCharacterSet]] componentsJoinedByString:@""];

NSString *PKK = [[[NSString stringWithFormat:@"%s",pub_key] componentsSeparatedByCharactersInSet:[NSCharacterSet newlineCharacterSet]] componentsJoinedByString:@""];
NSString *pubK = [[PKK dataUsingEncoding:NSUTF8StringEncoding] base64EncodedString];
NSString *publicKey = [[pubK componentsSeparatedByCharactersInSet:[NSCharacterSet newlineCharacterSet]] componentsJoinedByString:@""];


//IMP: publicKey is shared along with encrypted text(HELLO string encrypted with PrivateKey)

I am encrypting HELLO with PrivateKey using the below:

NSString *myString = @"HELLO";
const char *msg = (const char *)[myString cStringUsingEncoding:NSASCIIStringEncoding];;
err = malloc(130);
if((encrypt_len = RSA_private_encrypt(strlen(msg), (unsigned char*)msg, (unsigned char*)encrypt, rsaKeyPair, RSA_PKCS1_PADDING)) == -1) {
    ERR_load_crypto_strings();
    ERR_error_string(ERR_get_error(), err);
    fprintf(stderr, "Error encrypting message: %s\n", err);
}
NSString *validatorBase64 = [[[NSString stringWithFormat:@"%s",encrypt] dataUsingEncoding:NSUTF8StringEncoding] base64EncodedString];
NSString *validator = [[validatorBase64 componentsSeparatedByCharactersInSet:[NSCharacterSet newlineCharacterSet]] componentsJoinedByString:@""];

Answer 1

You have an issue on this line:

NSString *validatorBase64 = [[[NSString stringWithFormat:@"%s",encrypt] dataUsingEncoding:NSUTF8StringEncoding] base64EncodedString];

because encrypt is binary bytes and not a null-terminated string. This is likely what is causing your variable length results.

Here's a short program in pure C that uses RSA_private_encrypt/RSA_public_decrypt:

#include <stdio.h>
#include <string.h>
#include <openssl/bio.h>
#include <openssl/evp.h>
#include <openssl/rsa.h>

int main() {
   BIGNUM *e = BN_new();
   BN_set_word(e, 65537);

   RSA *rsaKeyPair = RSA_new();
   RSA_generate_key_ex(rsaKeyPair, 2048, e, NULL);
   BN_free(e);

   const unsigned char *plaintext = (unsigned char *)"HELLO";
   unsigned char crypttext[RSA_size(rsaKeyPair)];

   const int cryptLength = RSA_private_encrypt(
      strlen((const char *)plaintext), plaintext,
      crypttext,
      rsaKeyPair,
      RSA_PKCS1_PADDING);
   printf("encrypted length: %d\n", cryptLength);
   printf("encrypted data: ");
   for (int i = 0; i < cryptLength; ++i)
      printf("%02x", crypttext[i]);
   printf("\n");

   printf("encrypted data (base64): ");
   BIO *base64 = BIO_new(BIO_s_mem());
   base64 = BIO_push(BIO_new(BIO_f_base64()), base64);
   BIO_write(base64, crypttext, cryptLength);
   BIO_flush(base64);
   char *base64Data;
   const long base64Length = BIO_get_mem_data(base64, &base64Data);
   for (int i = 0; i < base64Length; ++i)
      printf("%c", base64Data[i]);
   printf("\n");
   BIO_free(base64);

   unsigned char decrypted[RSA_size(rsaKeyPair)];
   const int plainLength = RSA_public_decrypt(
      cryptLength, crypttext,
      decrypted,
      rsaKeyPair,
      RSA_PKCS1_PADDING);
   printf("decrypted length: %d\n", plainLength);

   decrypted[plainLength] = 0;
   printf("decrypted data: %s\n", (const char *)decrypted);

   RSA_free(rsaKeyPair);

   return 0;
}

Sample output:

$ ./rsa
encrypted length: 256
encrypted data: 2a884725dacdb961a51db22444b23a7802c5c612038ad8067bfe8b2db50e2c110fa2fc198ead4db314b9af57ada233228b7f07e09f821dd1928f2358f337bafa6915ae1f394b787a2250f19ff9e8babf9ffce0d7efebff95be5e017225223c05f8d3f93fa1126a9e77d485b38d01bbdf041fece43a388855695f9acd150f968aa23d0f7c247339f9953074171ad168cb06f2b6ff1c59dbde687a97da4360f0883b2a4d399b5213d3dee9a061ad0335f711acbecb212bc8ec1b5c2a3f9dbfc7d695c3593dc634b8b32727c7072cdcc716dfa2e86732fd54dfdbb193c0b0e0cb6d81f408cc12c4b97308c166dfbb0c8934dcba92d2e528c994ed9f10ec44d51ecb
encrypted data (base64): KohHJdrNuWGlHbIkRLI6eALFxhIDitgGe/6LLbUOLBEPovwZjq1NsxS5r1etojMi
i38H4J+CHdGSjyNY8ze6+mkVrh85S3h6IlDxn/nour+f/ODX7+v/lb5eAXIlIjwF
+NP5P6ESap531IWzjQG73wQf7OQ6OIhVaV+azRUPloqiPQ98JHM5+ZUwdBca0WjL
BvK2/xxZ295oepfaQ2DwiDsqTTmbUhPT3umgYa0DNfcRrL7LISvI7BtcKj+dv8fW
lcNZPcY0uLMnJ8cHLNzHFt+i6Gcy/VTf27GTwLDgy22B9AjMEsS5cwjBZt+7DIk0
3LqS0uUoyZTtnxDsRNUeyw==

decrypted length: 5
decrypted data: HELLO

Answer 2

@rhashimoto: u really helped me.. Now I get a fix length encrypted string..

But the base64 generated doesn't match on server, might be I am forwarding the public key in wrong way..

-(void)generateKeyPairs{
RSA *rsaKeyPair = NULL;
EVP_PKEY *PrivateKey = NULL;
rsaKeyPair = RSA_new();

BIGNUM *e = NULL;
e = BN_new();
BN_set_word(e, 65537);

RSA_generate_key_ex(rsaKeyPair, 2048, e, NULL);


// Now we need a private key object
PrivateKey = EVP_PKEY_new();

BIO *pri = BIO_new(BIO_s_mem());
BIO *pub = BIO_new(BIO_s_mem());

PEM_write_bio_RSAPrivateKey(pri, rsaKeyPair, NULL, NULL, 0, NULL, NULL);
PEM_write_bio_RSAPublicKey(pub, rsaKeyPair);

size_t pri_len = BIO_pending(pri);
size_t pub_len = BIO_pending(pub);

char *pri_key = malloc(pri_len + 1);
char *pub_key = malloc(pub_len + 1);

BIO_read(pri, pri_key, pri_len);
BIO_read(pub, pub_key, pub_len);

pri_key[pri_len] = '\0';
pub_key[pub_len] = '\0';

NSString *PK = [[[NSString stringWithFormat:@"%s",pri_key] componentsSeparatedByCharactersInSet:[NSCharacterSet newlineCharacterSet]] componentsJoinedByString:@""];
NSString *privateKey = [self RemoveKeyLabelFrom:[NSString stringWithFormat:@"%@",PK]];

NSString *PKK = [[[NSString stringWithFormat:@"%s",pub_key] componentsSeparatedByCharactersInSet:[NSCharacterSet newlineCharacterSet]] componentsJoinedByString:@""];
NSString *publicKey = [self RemoveKeyLabelFrom:[NSString stringWithFormat:@"%@",PKK]];

NSLog(@"Private Key: %@",privateKey);
NSLog(@"Public Key: %@",publicKey);

}


- (NSString *)RemoveKeyLabelFrom:(NSString *)str {
str = [str stringByReplacingOccurrencesOfString:@"-----BEGIN RSA PRIVATE KEY-----" withString:@""];
str = [str stringByReplacingOccurrencesOfString:@"-----END RSA PRIVATE KEY-----" withString:@""];
str = [str stringByReplacingOccurrencesOfString:@"-----BEGIN RSA PUBLIC KEY-----" withString:@""];
str = [str stringByReplacingOccurrencesOfString:@"-----END RSA PUBLIC KEY-----" withString:@""];

return str;
}

Please tell me wht am I missing to get the proper public key..