I'm currently developing a system to transmit data between client and server, and was wondering what the strength of the encryption I planned to use was.
My thought was to have a private/public RSA key pair and hand out the public key to each client (leaving the private key solely on the server). Each client would then generate their own AES key and RSA encrypt it. They would then AES encrypt their data and send the encrypted data and encrypted AES key to the server. The server would then decrypt the AES key using the private key, and then decrypt the data using the AES key.
Are there any security flaws I am missing and/or is there a better way to do this?
This is almost exactly how SSL/TLS works (check out the handshake section). The only thing to make it stronger is to generate the AES key for each connection, rather than using the same key each time. You might also want to digitally sign messages that go back and forth to avoid man-in-the-middle and other spoofing attacks.
Generally speaking, creating a good cryptosystem is very difficult. When possible, you should always favor an existing (trusted) application to help out. In this case, you might consider sending your messages using HTTPS, rather than creating your own system.
You should give us more information about the language and platform you are using, so that we can give you specific recommendations about libraries that already exist and wich will handle the details for you. Using cryptographic primitives directly is not trivial and difficult to get exactly right, and with cryptography, you have to be "not exactly right" only once for your security to be broken.
To answer your question, it's generally a better idea to create a session secret (the AES key) through a Diffie-Hellman exchange, and each side use its private RSA key to sign its key-exchange data.
Otherwise, if the session secret is not established through a DH exchange, an adversary that gains access to the private RSA key (which has to be stored somewhere) could decrypt all traffic that was ever sent between the client and server.
If the secret is established through a DH exchange, then only the authentication part of the protocol would be exposed. Although an adversary in possession of the private RSA key would then not be able to read any previous communication, he still could either enter an authenticated dialog with the client/server or launch a man-in-the-middle attack (which may or may not be easily done, depending on the network).
One vulnerability would be if an attacker substituted their public key for the "real" public key. Then they would be able to intercept traffic as a "man-in-the-middle."
Protocols like S/MIME, PGP, and TLS use RSA encryption to transport keys just as you propose. However, the public keys they use are in certificates signed by trusted authorities.
The integrity of these trusted authorities must be carefully protected. For example, they might be burned into a tamper-proof hardware token, or a MAC might be computed over them using a password.
I'm not sure your approach will protect anything! You're going to keep the private key on the server and the public key will be distributed. If I get a hold of your public key, I'll be able to interact with your server at will. I think you should reverse the key ownership; each client will hold it's-own private key and the server will have a list of public keys (ala SSH). The server will have to add 'authorized' public keys and only to holders of the private keys will be able to connect.
Hope this helps.
