I'm working on a pet project that will (eventually, when it's done) allow for secure file transfers (there's more to it than just that, but the rest isn't particularly relevant). I'd like to use the OpenSSL library, since it seems to be the most complete free cryptography library (and I need support for basic symmetric encryption and hashing, in addition to SSL/TLS).
I'm looking to implement a security scheme similar to SSH. Basically, a user would connect to my computer with TLSv1 (SSLv3.1). I'd like the connection to succeed regardless of security. Then, I want to be able to inspect the public key (not an entire certificate) that the user used. That key would be compared against known public keys, and if it matched, then the user would be allowed to access a certain set of commands. If it didn't match, the user would have the option to use the connection to apply to have his/her public key added to my collection, but other than that would not be able to access my services.
I don't have any particular need for certificates here. It would be much simpler for me if I could just skip all the certificate details and work only with the raw encryption keys. This is because this model follows a web-of-trust model, not the hierarchical model used by most SSL/TLS connections, so I don't need any CA's or signed certificates.
Unfortunately, the documentation of most of OpenSSL is, well, nonexistent. All the relevant articles I find seem to be occupied with setting up a "standard" SSL/TLS connection, where the server's certificate is verified all the way up to a set of root certificates. This can be useful, but it's hard for me to figure out how to get these non-traditional SSL connections up and running.
Can anyone suggest any articles or documentation that might help me figure out how to accomplish this?
(The use of OpenSSL is not set in stone, and I could switch to another library if it provides a better way of accomplishing this, as well as hashing [SHA-512] and symmetric encryption [AES]. I'm aiming at targeting Linux, but it would be nice if the final product was portable to Windows so my friends could use it too.)
To expand on Eugene's answer (I would have put this as a comment, but it's a bit long)...
Having done this sort of things with the FOAF+SSL project (later renamed WebID), sticking with X.509 certificates makes the implementation easier, simply because most SSL/TLS stacks are designed with them in mind (and their API reflect this).
Last time I checked FOAF+SSL, the traditional PKI checks were still in place for the client to check the server certificate. Another option, similar to SSH, would be to accept the public key/certificate the first time you encounter it and warn the user when it changes. That's more or less the way SSH works anyway (in particular, I guess that few people actually check the key's fingerprint out of bands the first time they see it).
Considering just the client-certificate usage (although some of this could apply to server certs in a similar way):
- Most server libraries seem to be able to process X.509 certificates, but let you change the way they are verified (e.g.
X509TrustManager in Java).
- Whilst you won't be able to trust anything the client-cert says until you have verified it otherwise, being able to embed some extra information (such as a Subject DN or Subject Alternative Name to see who the user claim to be) can help (a) the users organise their certs and (b) give a hint for the verifier to know what to look for. A bare public key can be hard to manage.
- A number of existing client tools (especially browsers) use X.509 certificates when doing SSL/TLS client authentication. Not much needs to be done to configure a client to use a self-signed X.509 cert (as opposed to a cert from a PKI). (There are very few tools that support OpenPGP for TLS, I'm not sure any are able to use it as a form of client certificate.)
- Since you won't be able to trust the cert without external checks, it doesn't matter whether it's self-signed or not (i.e. whether the issuer and the subject are the same), at least assuming the user wouldn't send you a cert with which it wouldn't agree (so it wouldn't have to be sealed by its own key). A consequence of that is that you can build a service to issue certs quite easily. In-browser key-generation, for example, is convenient for users who don't want to use
openssl or keytool commands. Here is an example service that will issue a certificate with the SAN the user wants (there might be more recent versions if you check with the FOAF+SSL/WebID project). Whichever private key or issuer name such a service uses barely matters, but since browsers are designed around traditional PKIs, it doesn't make it easy to use really self-signed certificates.
There are also issues when it comes to asking for a specific client-certificate. The TLS 1.1 specification explicitly allows empty certification authorities (see RFC 4346), whereas the TLS 1.0 were silent on the subject. In practice, even with TLS 1.0, most client tools seem to be happy with an empty list (they'll just offer more choice). If you want your certificates for your system to be easily identifiable, you could use the same issuer DN for all these certs, even if they're not signed with the same private key in practice (again, since you would ignore the signature).
Use self-signed certificates - this is the same as "raw" keys but easier to manage (see this question regarding how to accept or not accept a self-signed certificate in openssl).
