I am building an application and I am planning on using OpenSSL for securing data transfers.
I am planning on only having the client validate the server's certificate. I am confused on how I should secure the server's certificate. I would like to encrypt the server's certificate containing the private key, but I do not want to use any hard coded keys for this encryption.
What are some of the common practices followed by applications employing SSL?
You don't have to secure the server's certificate. It's a public document
The server's certificate does not contain the private key.
You don't need to do any encryption. The only server asset you need to protect is its private key, which is entirely distinct from its certificate. In the case of OpenSSL it can even be a different file.
not quite sure what you're trying to ask. the server cert is sent to you, the client; you validate the cert by checking its signature (use SHA-1 not MD5, MD5 has been cracked.) The key you have from the CA is the public side; the CA and the server cert holder keep their private keys to themselves. You can validate the cert because the public key is enough to decrypt a message that has been encrypted with the private key. So you don't have to worry, on the cient side, about keeping the cert encrypted at all.
Have a look at the Wikipedia article on SSL/TLS.
Just to make sure we have our terminology straight, an "SSL certificate" in common parlance is really composed of two components:
The public certificate component is signed by your chosen CA (certificate authority), after which it can be freely distributed. It does not need to be secured or encrypted, and indeed it will be sent to clients that connect to your server as part of the SSL negotiation.
The private key component should be protected. In the majority of cases, this is simply stored as an encrypted file on the server. Upscale solutions use dedicated "tamperproof" crypto hardware (HSMs -- hardware security modules) to store the private key. These range from smart-card based solutions to multi-key, network enabled appliances with m/n controls etc etc. There are risks (not to mention costs) associated with HSMs that I will not go into here.
Many applications simply retain the private key on disk. There are a couple of options to secure the key file:
As always, there is a security trade-off. In particular, if you are using password-protected encryption on the private key file and you experience an unexpected application restart (eg power outage), then somebody will need to be available to provide the password to the app as it restarts. Storing the password in a file that is read by system initialization scripts (as encouraged by at least two web server vendors) adds little in terms of real security. It's hard to recommend leaving the private key file unencrypted but if you are the sole admin/techy in a small shop, you should definitely consider what might happen if the server reboots when you are not available, and what the costs might be to your business.