I was thinking about communication between two computers using public-key encryption.
Let us assume that on one computer side I generate a private key and a public key. If I send the public key to the other computer, can it still be captured by a packet sniffer and use it to decrypt my message? If yes, what can be done to prevent this and what is the use of public key encryption then?
Update
Is SSL equivalent to public-key encryption? Should I use it instead?
No. The public key is not used to decrypt, but to encrypt. You decrypt with your private key. That way, only you can decrypt the message.
Usually though, private-public key encryption is too expensive to encrypt data with. You'd use a private/public encryption scheme to exchange a shared key (large random number) to encrypt your data with.
Example: Alice creates a private/public keypair, sends the public key to Bob. Bob creates a large random number and encrypts it with Alice's public key. Alice can decrypt that and find the large random number. Alice and Bob then use the random number to encrypt their data stream.
As additional security, you could change keys periodically.
To update on your SSL question: it works exactly as I describe above. See also http://en.wikipedia.org/wiki/Transport_Layer_Security
The answer is yes, it can be captured by sniffer.
Why RSA alone does not fit with Matt's situation?
In Matt's situation, he want both of 2 computers to be ensured legal. My point is RSA can only ensure 1 of them legal, instead of both of them.
Another important thing is cracker CAN also use private key to encrypt and public key to decrypt. The key can make client know it was connecting to a "legal" server, but can't help server to ensure it's a "legal" client since key stored at client side or send from server can be leaked. A better solution is implement your own hash algorithm to prevent being hacked.
Here is an article about RSA private key encryption:
http://www.codeproject.com/Articles/38739/RSA-Private-Key-Encryption
Let's think about bank website, the client should know he is on legal website, but bank server doesn't need to ensure the client is legal or not since mobile phone authentication and other methods can do the trick. The mobile phone authentication is just something like "hash method", so, hash method is a must. RSA alone can not ensure safety.
Without hash method, if a cracker gets key stored, or transmitted by Internet, he can easily make a fake client without any difficulties.
Well, then what's your solution?
Since I'm implementing a Client-Server based software, I can share you my solution:
I saved private key into Client's source code.
When server send some response or client send some request, use private/public key to encrypt and decrypt by the other side.
You need some protection, like code obfuscation to protect the key stored in client.
You need to design an hash algorithm to ensure data sent by client is legal. The hacker might get your key finally, but hard to know what your algorithm is so it's still safe enough.
Hash algorithm means an algorithm combined with add some salt/SHA-1/UUID/timestamp...etc. I don't mean you should invent a new encrypt algorithm.
For example, if the plain text is
I'm so awesomeThe algorithm can be:
Result = SHA1(salt + plain text + timestamp + anything you like)
If find your algorithm leaked, just change some key values.
What if your hash algorithm leaked?
Remember there's no algorithm can not be hacked. We don't need to build a castle can not be destroyed, we just need to make out enemy pay hard.
Still, you also need a quick "big red button" if anything goes wrong. Hash algorithm can play this role quite easy, a small modification can make crackers pay lots of time to hack it out. That's already good enough.
UPDATE:
Is SSL equivalent to public-key encryption? Should I use it instead?
Yes, but you still have to store key somewhere better than send it through network. HTTPS/TLS is another decent choice.
