How to troubleshoot SSL “bad record MAC” exception

Question

I am using an Apache CXF client, running in a Windows Java 1.6.0_29-b11 VM to connect to an IBM mainframe (I believe it is zSeries), and invoking a SOAP Web Service running there. The connection is done through SSL/TLS, and most of the time works fine.

However, from time to time I have SSL Exceptions with a bad record MAC message. Here is the output of the program using with the javax.net.debug property.

2011-11-16 12:32:37,731 INFO  LoggingOutInterceptor: Outbound Message
---------------------------
ID: 29
Address: https://1.2.3.4/access/servlet/blabla.atk123
Encoding: UTF-8
Content-Type: text/xml
Headers: {Accept=[*/*], SOAPAction=["Blablaaction/ATK123.Execute"]}
Payload: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><ATK123.Execute xmlns="Blabla"><Usrid>WA</Usrid><Usrpwd>54321</Usrpwd><Ultautid>9999</Ultautid></ATK123.Execute></soap:Body></soap:Envelope>
--------------------------------------
pool-1-thread-1, setSoTimeout(30000) called
pool-1-thread-1, WRITE: TLSv1 Application Data, length = 321
pool-1-thread-1, WRITE: TLSv1 Application Data, length = 262
pool-1-thread-1, READ: TLSv1 Application Data, length = 483
pool-1-thread-1, READ: TLSv1 Application Data, length = 16148
pool-1-thread-1, READ: TLSv1 Application Data, length = 282
%% Invalidated:  [Session-1, SSL_RSA_WITH_RC4_128_SHA]
pool-1-thread-1, SEND TLSv1 ALERT:  fatal, description = bad_record_mac
pool-1-thread-1, WRITE: TLSv1 Alert, length = 22
pool-1-thread-1, called closeSocket()
pool-1-thread-1, handling exception: javax.net.ssl.SSLException: bad record MAC
2011-11-16 12:32:38,511 WARN  PhaseInterceptorChain: Interceptor for {Blabla}ATK123#{Blabla}Execute has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: bad record MAC
    at org.apache.cxf.interceptor.LoggingInInterceptor.logging(LoggingInInterceptor.java:144)
    at org.apache.cxf.interceptor.LoggingInInterceptor.handleMessage(LoggingInInterceptor.java:73)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:263)
    at org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:797)
.... (more stuff)

Unfortunately, I don't have possibilities to modify or debug the endpoint at the server.

What could be causing this?

How can I isolate and fix this behavior?

Answer 1

This isn't related to Java, it is an SSL/TLS failure:

20  Bad record MAC  fatal   Possibly a bad SSL implementation, or payload
                            has been tampered with e.g. FTP firewall rule
                            on FTPS server.

It probably has something to do with the SSL implementation and the amount of data that is being sent being too big, I doubt it is random.

Answer 2

If you are getting a lot of bad packets due to some fault in your network it can happen that a bad packet will at random survive the 32-bit TCP checksum. About 1 in 4 billion bad packets will slip by TCP. Once this packet is delivered to SSL it will generate a bad record MAC for sure, because the SSL Mac is 96 bits in size.

If this is the cause, the only solution is improve the network.

Note that, in general, this is a very unlikely cause of a bad record MAC. Even a network with faulty hardware that generates bad packets is unlikely to generate them with correct IP and TCP metadata such that the packets are actually passed to the socket corresponding to the TLS connection.