How to generate password for RSA / AES encryption

I want to send large data encrypted with RSA through sockets. I use openssl and c.

Because RSA decryption is quite slow I use the common and straight forward way to encrypt the data with AES first, and afterwards I encrypt the used AES password with RSA. Then I send both, the AES encrypted data and the RSA encrypted password, through the socket and do the encryption the other way around.

I do the AES encryption with:

EVP_CIPHER_CTX en;
unsigned char password[65];
int i, x = 0;
unsigned char key[32], iv[32];
unsigned char *ciphertext;

i = dataLength + AES_BLOCK_SIZE -1;
ciphertext = (unsigned char *)malloc(i);

EVP_CIPHER_CTX_init(&en);
EVP_EncryptInit_ex(&en, EVP_aes_256_cbc(), NULL, key, iv);
EVP_EncryptUpdate(&en, ciphertext, &i, (unsigned char*)data, dataLength);
EVP_EncryptFinal_ex(&en, ciphertext+i, &x);

But how do I create the key and the iv securely? Right now I Use the following function:

EVP_BytesToKey(EVP_aes_256_cbc(), EVP_sha1(), salt, password, 64, 9, key, iv);

My question is: How do I create "password" correctly?

Because if I use rand() or something equal my attempt was completely useless because anybody who is able to get behind the "randomness" used for the "password" generation is able to decrypt the data anyway without caring about the RSA encryption of the "password".

Is there a function for secure passwordgeneration in openssl? Or is EVP_BytesToKey() just the wrong way to do what I want to do?

The default RAND_bytes method is fortunately seeded per thread, and by default uses the random number generator available from the operating system. The OpenSSL documentation seems to be out of date where Windows is involved, but you can find more information on this by looking at the answer of the venerable Thomas Pornin on security.stackoverflow.com.

EVP_BytesToKey is used to generate keys from passwords. EVP_BytesToKey is a key derivation function (KDF) that is specific to OpenSSL. OpenSSL also implements PBKDF2 which is the NIST approved method of password based key derivation function (PBKDF). But as you want a random key, not a derived key, none of those functions apply.

So please use rand(). If possible, try to check how the function is seeded for your specific platform.

Also note OpenSSL 1.1.0c changed the digest algorithm used in some internal components. Formerly, MD5 was used, and 1.1.0 switched to SHA256. Be careful the change is not affecting you in both EVP_BytesToKey and commands like openssl enc.