Which attacks are possible concerning my security

Question

Despite all the advices to use SSL/https/etc. I decided to implement my own security layer on top of http for my application... The concept works as follows:

User registers -> a new RSA Keypair is generated
the Private Key gets encrypted with AES using the users login Password
(which the server doesnt know - it has only the sha256 for authentication...)

Server stores the hash of the users password
 and the Encrypted Private Key and Public Key

User logs in -> authenticates with nickname+password hash
(normal nick/password -> IP-bound sessionid authentication)
Server replies: sessionid, the Encrypted RSA Private Key
    and an Encrypted randomly generated Session Communication Password

Client decrypts the RSA Private Key with the users Password
Client decrypts the Session Communication Password with the RSA Private Key

---> From this point on the whole traffic gets AES-encrypted
     using that Session Password

I found no hole in that chain - neither the private key nor the login password get ever sent to the server as plaintext (I make no use of cookies, to exclude the possibility of the HTTP Cookie header to contain sensitive information)... but I am biased, so I ask - does my security implementation provide enough... security?

Answer 1

I'm not a crypto or security expert by any means, but I do see one serious flaw:

There is no way the client can know that it is running the right crypto code. With SSL/TLS there is an agreed upon standard that both your browser vendor and the server software vendor have implemented. You do not need to tell the browser how SSL works, it comes built in, and you can trust that it works correctly and safely. But, in your case, the browser only learns about the correct protocol by receiving plain-text JavaScript from your server.

This means that you can never trust that the client is actually running the correct crypto code. Any man-in-the-middle could deliver JavaScript that behaves identically to the script you normally serve, except that it sends all the decrypted messages to the attacker's servers. And there's no way for the client to protect against this.

That's the biggest flaw, and I suspect it's a fatal flaw for your solution. I don't see a way around this. As long as your system relies on delivering your crypto code to the client, you'll always be susceptible to man-in-the-middle attacks. Unless, of course, you delivered that code over SSL :)

Answer 2

Why does everyone have to come up with their secure transport layer? What makes you think you've got something better than SSL or TLS? I simply do not understand the motivation to re-invent the wheel, which is a particularly dangerous thing to do when it comes to cryptography. HTTPS is a complex beast and it actually does a lot of work.

Remember, HTTPS also involves authentication (eg: being able to know you are actually talking to who you think you are talking to), which is why there exists a PKI and browsers are shipped with Root CA's. This is simply extremely difficult (if not impossible) to re-invent and prone to security holes. To answer you question, how are you defending against MITM attacks?

TLDR: Don't do it. SSL/TLS work just fine.

/endrant.

Answer 3

Javascript encryption can be enough when your adversary is an eavesdropper that can see your traffic but not modify it.

Please note that I am not referring to your specific idea (which I did not take the time to fully understand) but to the general concept of Javascript encryption.

Answer 4

The major problem you have is that your client crypto code is delivered as Javascript over unauthenticated HTTP.

This gives the Man-In-The-Middle plenty of options. He can modify the code so that it still authenticates with your server, but also sends the password / private key / plaintext of the conversation to him.

Answer 5

It looks like you've made more complexity than is needed, as far as "home-grown" is concerned. Specifically, I see no need to involve assymetric keys. If the server already knows the user's hashed password, then just have the client generate a session id rolled into a message digest (symmetrically) encrypted via the client's hashed password.

The best an attacker might do is sniff that initial traffic, and attempt a reply attack...but the attacker would not understand the server's response.

Keep in mind, if you don't use TLS/SSL, then you won't get hardware-accelerated encryption (it will be slower, probably noticeably so).

You should also consider using HMAC, with the twist of simply using the user's password as the crypto key.

Answer 6

SSL/TLS provide transport layer security and what you've done does nothing but do that all over again for only the authorization process. You'd be better served to focus on authorization techniques like client certificates than to add an additional layer of line-level encryption. There's a number of things you could also introduce that you haven't mentioned such as encrypted columns in SQL Server 2008, IPSec, layer 4 & 7 hardware solutions and even setting up trusts between the server and client firewalls. My biggest concern is how you've created such a deep dependency on the username and password, both which can change over time in any system.

I would highly recommend that you reconsider using this approach and look to rely on more standard techniques for ensuring that credentials are never stored unencrypted on the server or passed in the clear from the client.