RabbitMQ: handshake error when attempting to use S

Question

I am trying to use SSL certificates with RabbitMQ but I keep getting handshake errors with the broker.

The certificates that I have generated work fine when using the openssl 's_client' and 's_server' commands in separate terminal windows and utilizing port 8443 as detailed in the SSL Troubleshooting guide (http://www.rabbitmq.com/troubleshooting-ssl.html).

The problem appears when I attempt to connect to the RabbitMQ SSL port 5671 using the same openssl 's_client' command:

Running this:

openssl s_client -connect localhost:5671 -cert /etc/rabbitmq/ssl/client/cert.pem -key /etc/rabbitmq/ssl/client/key.pem -CAfile /etc/rabbitmq/ssl/certificate_auth/cacert.pem

Produces this:

CONNECTED(00000003)
depth=1 CN = RMQCA
verify return:1
depth=0 CN = roger.xxxxxx.com, O = server
verify return:1
139997248210760:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1256:SSL alert number 40
139997248210760:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake  failure:s23_lib.c:177:
---

The SSL listener starts fine as indicated in the RabbitMQ log:

=INFO REPORT==== 19-May-2014::15:45:34 ===
 started TCP Listener on [::]:5672

=INFO REPORT==== 19-May-2014::15:45:34 ===
 started SSL Listener on [::]:5671

When attempting to connect to port 5671 with 's_client' the error appears:

=INFO REPORT==== 19-May-2014::17:20:39 ===
accepting AMQP connection <0.3263.0> ([::1]:58538 -> [::1]:5671)

=ERROR REPORT==== 19-May-2014::17:20:39 ===
SSL: certify: ssl_handshake.erl:1346:Fatal error: handshake failure

=ERROR REPORT==== 19-May-2014::17:20:44 ===
error on AMQP connection <0.3263.0>: {ssl_upgrade_error,
                                      {tls_alert,"handshake failure"}} (unknown POSIX error)

RabbitMQ Config file:

[    
    {rabbit, [
      {ssl_listeners, [5671]},
      {ssl_options, [{cacertfile, "/etc/rabbitmq/ssl/certificate_auth/cacert.pem"},
                     {certfile, "/etc/rabbitmq/ssl/server/cert.pem"},
                     {keyfile, "/etc/rabbitmq/ssl/server/key.pem"},
                     {verify, verify_peer},
                     {fail_if_no_peer_cert, false}]}
     ]} 
].

RabbitMQ info:

[{pid,10375},
 {running_applications,
     [{rabbitmq_management,"RabbitMQ Management Console","3.2.3"},
      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.2.3"},
      {webmachine,"webmachine","1.10.3-rmq3.2.3-gite9359c7"},
      {mochiweb,"MochiMedia Web Server","2.7.0-rmq3.2.3-git680dba8"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.2.3"},
      {rabbit,"RabbitMQ","3.2.3"},
      {ssl,"Erlang/OTP SSL application","5.3.3"},
      {public_key,"Public key infrastructure","0.21"},
      {crypto,"CRYPTO version 2","3.2"},
      {asn1,"The Erlang ASN1 compiler version 2.0.4","2.0.4"},
      {os_mon,"CPO  CXC 138 46","2.2.14"},
      {inets,"INETS  CXC 138 49","5.9.8"},
      {mnesia,"MNESIA  CXC 138 12","4.11"},
      {amqp_client,"RabbitMQ AMQP Client","3.2.3"},
      {xmerl,"XML parser","1.3.6"},
      {sasl,"SASL  CXC 138 11","2.3.4"},
      {stdlib,"ERTS  CXC 138 10","1.19.4"},
      {kernel,"ERTS  CXC 138 10","2.16.4"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang R16B03-1 (erts-5.10.4) [source] [64-bit] [smp:2:2] [async-threads:30] [hipe] [kernel-poll:true]\n"},
 {memory,
     [{total,43812088},
      {connection_procs,5616},
      {queue_procs,42528},
      {plugins,451248},
      {other_proc,13805200},
      {mnesia,72752},
      {mgmt_db,10208},
      {msg_index,34560},
      {other_ets,1159472},
      {binary,1030272},
      {code,21819091},
      {atom,793505},
      {other_system,4587636}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,787819724},
 {disk_free_limit,50000000},
 {disk_free,31267266560},
 {file_descriptors,
     [{total_limit,924},{total_used,4},{sockets_limit,829},{sockets_used,2}]},
 {processes,[{limit,1048576},{used,215}]},
 {run_queue,0},
 {uptime,7893}]
...done.

Any help would be greatly appreciated

Thanks in advance.

UPDATE:

I get the following errors when trying to connect with the rabbitmqadmin utility.

Log File:

=INFO REPORT==== 20-May-2014::14:39:12 ===
accepting AMQP connection <0.16589.0> ([::1]:58922 -> [::1]:5671)

=ERROR REPORT==== 20-May-2014::14:39:12 ===
SSL: certify: ssl_handshake.erl:1346:Fatal error: handshake failure

=ERROR REPORT==== 20-May-2014::14:39:17 ===
error on AMQP connection <0.16589.0>: {ssl_upgrade_error,
                                       {tls_alert,"handshake failure"}} (unknown POSIX error)

The rabbitmqadmin command produced the following:

*** Could not connect: [Errno 1] _ssl.c:492: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

Answer 1

I worked my way through similar troubles (using RabbitMQ 2.7.1 / Erlang R14B04). Here's what I've found:

The RabbitMQ plugins page and at least one other site recommend enabling the plugin rabbitmq_auth_mechanism_ssl. If rabbitmq-plugins is an invalid command on your system, this page describes how to enable it on Ubuntu. (Apparently the apt-get package doesn't have quite the expected behavior on Debian-based systems.) Your output (from rabbitmqctl report, I presume) says you don't have rabbitmq_auth_mechanism_ssl enabled.

For your rabbitmq.config, you'll need to make sure "EXTERNAL" is listed as one of the auth_mechanisms. The line's syntax is {auth_mechanisms, ['PLAIN', 'AMQPLAIN', 'EXTERNAL']} and appears as one item in the default, "rabbit" portion of the configuration.

You should also make sure the certificate your client presents has the appropriate values set for both keyUsage and extendedKeyUsage, as RabbitMQ is more strict about these than s_server. For debugging / testing purposes, you may want to be extremely permissive with these. You can set the keyUsage in your openssl config. A broadly-acceptable openssl config may have lines like this

keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment, keyAgreement, keyCertSign, cRLSign extendedKeyUsage = 1.3.6.1.5.5.7.3.1, 1.3.6.1.5.5.7.3.2

(I think the .2 OID, "TLS Web Client Authentication" is important for connecting to RabbitMQ, but I haven't done careful tests.)

This will produce certificates with this block near the end:

X509v3 Key Usage: Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment, Key Agreement, Certificate Sign, CRL Sign X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication There should be more output from s_client. In particular, I'm interested in the final line, which should look something like "Verify return code: 0 (ok)" If you have a non-zero / error message there, post it and pivot off that in your searches. (#19 is surprisingly common, given that it's not really an error.)

When I got to this point, when I tried to make a simple pika.BlockingConnection, the handshake apparently completed just fine, but Rabbit removed EXTERNAL from the list specified in auth_mechanisms in the config. I confirmed I had rabbitmq_auth_mechanism_ssl enabled, but that by itself wasn't enough. (I discovered this by subclassing pika.credentials.ExternalCredentials and pass an instance as the "credentials" item in ConnectionParameters, adding a print start at the top of the subclass's response_for() method.) I fixed that by adding the following line to the rabbit portion of the config file, on the same level as ssl_listeners and ssl_cert_login_from:

{ssl_apps,[asn1,crypto,public_key,ssl]},

(I suspect newer versions of RabbitMQ turn that on by default, but my particular setup did not.)

If you've done all that and you're still having trouble, you might also try replacing "verify_peer" with "verify_none" in your RabbitMQ config. You probably don't want that in production, since it opens you up to anyone with a self-signed certificate, but it's another data point. Also, subclass the relevant things in pika and add in print statements to get more insight on what Rabbit's sending you and how your local client's interpreting it.

Answer 2

Here is the solution which worked for me:

Adding below ciphers in rabbitmq.config:

{ciphers, ["ECDHE-ECDSA-AES256-GCM-SHA384","ECDHE-RSA-AES256-GCM-SHA384",
                        "ECDHE-ECDSA-AES256-SHA384","ECDHE-RSA-AES256-SHA384", "ECDHE-ECDSA-DES-CBC3-SHA",
                        "ECDH-ECDSA-AES256-GCM-SHA384","ECDH-RSA-AES256-GCM-SHA384","ECDH-ECDSA-AES256-SHA384",
                        "ECDH-RSA-AES256-SHA384","DHE-DSS-AES256-GCM-SHA384","DHE-DSS-AES256-SHA256",
                        "AES256-GCM-SHA384","AES256-SHA256","ECDHE-ECDSA-AES128-GCM-SHA256",
                        "ECDHE-RSA-AES128-GCM-SHA256","ECDHE-ECDSA-AES128-SHA256","ECDHE-RSA-AES128-SHA256",
                        "ECDH-ECDSA-AES128-GCM-SHA256","ECDH-RSA-AES128-GCM-SHA256","ECDH-ECDSA-AES128-SHA256",
                        "ECDH-RSA-AES128-SHA256","DHE-DSS-AES128-GCM-SHA256","DHE-DSS-AES128-SHA256",
                        "AES128-GCM-SHA256","AES128-SHA256","ECDHE-ECDSA-AES256-SHA",
                        "ECDHE-RSA-AES256-SHA","DHE-DSS-AES256-SHA","ECDH-ECDSA-AES256-SHA",
                        "ECDH-RSA-AES256-SHA","AES256-SHA","ECDHE-ECDSA-AES128-SHA",
                        "ECDHE-RSA-AES128-SHA","DHE-DSS-AES128-SHA","ECDH-ECDSA-AES128-SHA",
                        "ECDH-RSA-AES128-SHA","AES128-SHA"]},
                  {fail_if_no_peer_cert,false}]}
    ]}
]

Answer 3

I had the same problem as @user3653959 and @Sarah Messer's answer lead me to the solution.

Your client certificate must have the TLS Web Client Authentication "X509v3 Extended Key Usage" attribute. Mine had only TLS Web Server Authentication due to an error in my client generation script.

To check your client certificate's capabilities, you can use the this command:

openssl x509 -noout -text -in client-certificate.pem

Then look for the "X509v3 extensions:" section and the "X509v3 Extended Key Usage:" subsection.

If you generate your client certificate using the example openssl.conf and client and server commands provided in the official "RabbitMQ - TLS Support" guide, it should work out of the box.

The key here is the extendedKeyUsage = 1.3.6.1.5.5.7.3.2 openssl config option in openssl.conf as @Sarah Messer points out. This is the "TLS Web Client Authentication" capability. OpenSSL s_server does not require this capability and that's why it works by default with it, but not with RabbitMQ. keyUsage = digitalSignature is enough as main usage options. Also, the "Common Name" (CN) of the client certificate is not important.

Just for reference

My environment:

• RabbitMQ 3.6.2
• Erlang 18.2
• Ubuntu 14.04.2 LTS (64-bit)
• Only TLSv1.2 enabled.

The error I was seeing in my RabbitMQ log:

=ERROR REPORT==== 21-Jun-2016::13:28:21 ===
SSL: certify: ssl_handshake.erl:1492:Fatal error: handshake failure

The error I was seeing via openssl s_client:

140735165813584:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:s3_pkt.c:1472:SSL alert number 40
140735165813584:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:s3_pkt.c:656: