Intermittent javax.net.ssl failure bad_record_mac

I have a Java Spring web app running on Tomcat through an Apache https proxypass which fails intermittently when it tries to access a secure IBM Watson service. Apache is secured with a LetsEncrypt cert, redirecting to Tomcat port 8080.

Environment:

Java: jdk1.7.0_80
Solaris 10
Tomcat 8.0.33
Apache 2.4.18

I turned on javax.net debug and I can see it gets through ServerHelloDone. Here is the rest of the log up to the exception.

ServerHelloDone
[read] MD5 and SHA1 hashes:  len = 4
0000: 0E 00 00 00                                        ....
ECDHClientKeyExchange
ECDH Public value:  { 4, 1, 15, 216, 52, 12, 28, 142, 36, 18, 118, 235, 237, 15, 74, 34, 33, 56, 45, 207, 106, 156, 33, 190, 210, 211, 175, 194, 89, 194, 183, 108, 144, 243, 51, 244, 179, 187, 96, 144, 10, 108, 146, 141, 244, 76, 71, 228, 210, 106, 52, 6, 241, 210, 130, 225, 126, 88, 210, 15, 184, 97, 206, 93, 65, 141, 34, 0, 3, 64, 70, 132, 192, 245, 110, 15, 18, 172, 202, 220, 232, 174, 254, 75, 160, 45, 48, 33, 180, 50, 39, 38, 122, 56, 13, 5, 15, 164, 61, 213, 135, 95, 115, 152, 246, 74, 157, 17, 26, 141, 90, 77, 105, 197, 209, 32, 163, 122, 22, 23, 152, 214, 79, 36, 192, 196, 109, 212, 137, 103, 125, 159, 17 }
[write] MD5 and SHA1 hashes:  len = 138
0000: 10 00 00 86 85 04 01 0F   D8 34 0C 1C 8E 24 12 76  .........4...$.v
0010: EB ED 0F 4A 22 21 38 2D   CF 6A 9C 21 BE D2 D3 AF  ...J"!8-.j.!....
0020: C2 59 C2 B7 6C 90 F3 33   F4 B3 BB 60 90 0A 6C 92  .Y..l..3...`..l.
0030: 8D F4 4C 47 E4 D2 6A 34   06 F1 D2 82 E1 7E 58 D2  ..LG..j4......X.
0040: 0F B8 61 CE 5D 41 8D 22   00 03 40 46 84 C0 F5 6E  ..a.]A."..@F...n
0050: 0F 12 AC CA DC E8 AE FE   4B A0 2D 30 21 B4 32 27  ........K.-0!.2'
0060: 26 7A 38 0D 05 0F A4 3D   D5 87 5F 73 98 F6 4A 9D  &z8....=.._s..J.
0070: 11 1A 8D 5A 4D 69 C5 D1   20 A3 7A 16 17 98 D6 4F  ...ZMi.. .z....O
0080: 24 C0 C4 6D D4 89 67 7D   9F 11                    $..m..g...
recipeorganizer.net-startStop-1, WRITE: TLSv1 Handshake, length = 138
[Raw write]: length = 143
0000: 16 03 01 00 8A 10 00 00   86 85 04 01 0F D8 34 0C  ..............4.
0010: 1C 8E 24 12 76 EB ED 0F   4A 22 21 38 2D CF 6A 9C  ..$.v...J"!8-.j.
0020: 21 BE D2 D3 AF C2 59 C2   B7 6C 90 F3 33 F4 B3 BB  !.....Y..l..3...
0030: 60 90 0A 6C 92 8D F4 4C   47 E4 D2 6A 34 06 F1 D2  `..l...LG..j4...
0040: 82 E1 7E 58 D2 0F B8 61   CE 5D 41 8D 22 00 03 40  ...X...a.]A."..@
0050: 46 84 C0 F5 6E 0F 12 AC   CA DC E8 AE FE 4B A0 2D  F...n........K.-
0060: 30 21 B4 32 27 26 7A 38   0D 05 0F A4 3D D5 87 5F  0!.2'&z8....=.._
0070: 73 98 F6 4A 9D 11 1A 8D   5A 4D 69 C5 D1 20 A3 7A  s..J....ZMi.. .z
0080: 16 17 98 D6 4F 24 C0 C4   6D D4 89 67 7D 9F 11     ....O$..m..g...
SESSION KEYGEN:
PreMaster Secret:
0000: 00 65 8C 39 8F 65 BD D3   AC EF B3 D8 AF 7D 62 CA  .e.9.e........b.
0010: DE 38 F7 FC B3 68 CD 17   4B A9 2E E3 57 76 E6 ED  .8...h..K...Wv..
0020: 44 C3 39 47 9C 6D 9B 11   EE 36 01 50 78 DC E2 6B  D.9G.m...6.Px..k
0030: 79 EF A2 86 B8 5A D2 0F   84 F8 48 23 3D 18 57 9E  y....Z....H#=.W.
0040: 59 FB                                              Y.
CONNECTION KEYGEN:
Client Nonce:
0000: 58 44 7E ED EE F3 E8 DA   8A 53 52 00 3B 11 2B E2  XD.......SR.;.+.
0010: 7D 41 F8 C0 0D CD BB C7   7D 97 32 CB BA 5D BF 22  .A........2..]."
Server Nonce:
0000: 54 AC E4 66 B7 28 95 0B   60 02 C3 3B 6C A8 D7 4E  T..f.(..`..;l..N
0010: 89 B6 26 85 3D 38 00 1A   2F 3F B1 4B D1 57 40 8B  ..&.=8../?.K.W@.
Master Secret:
0000: EF A0 29 3D 09 06 17 55   DB B9 96 5E 0F E4 58 86  ..)=...U...^..X.
0010: 38 42 88 15 DA 86 95 91   6B 6E 06 8C 33 D3 3D 09  8B......kn..3.=.
0020: D3 7E 1C E6 D6 10 53 7A   B2 A4 B7 E0 4A FB E0 BA  ......Sz....J...
Client MAC write Secret:
0000: 89 E4 9D 64 8B 81 87 CB   F8 4B 2B 18 49 08 E7 0E  ...d.....K+.I...
0010: D5 E7 69 DB                                        ..i.
Server MAC write Secret:
0000: F2 6A 0D C4 9B 2C 4D F2   64 9F DC DD 42 7A 67 4F  .j...,M.d...BzgO
0010: 25 7C 0E 4C                                        %..L
Client write key:
0000: C8 BD B9 72 CB CE 7B 09   40 E1 98 7D 8E 65 43 F3  ...r....@....eC.
Server write key:
0000: E9 E7 11 F1 FB BB AD 02   A1 43 BA 60 8F C2 68 83  .........C.`..h.
Client write IV:
0000: E2 A3 08 8D 1E A8 2B 64   DA B5 C2 8D F3 50 E9 59  ......+d.....P.Y
Server write IV:
0000: B5 3C 89 77 2E 4E 17 D5   74 25 3A 2A 92 6E D9 8B  .<.w.N..t%:*.n..
recipeorganizer.net-startStop-1, WRITE: TLSv1 Change Cipher Spec, length = 1
[Raw write]: length = 6
0000: 14 03 01 00 01 01                                  ......
Finished
verify_data:  { 118, 161, 158, 7, 159, 29, 185, 175, 148, 167, 176, 216 }
[write] MD5 and SHA1 hashes:  len = 16
0000: 14 00 00 0C 76 A1 9E 07   9F 1D B9 AF 94 A7 B0 D8  ....v...........
Padded plaintext before ENCRYPTION:  len = 48
0000: 14 00 00 0C 76 A1 9E 07   9F 1D B9 AF 94 A7 B0 D8  ....v...........
0010: 9C A6 C7 9D 1A D7 DC B9   96 A2 83 CA 9A 50 BD 3D  .............P.=
0020: 7D D9 72 4A 0B 0B 0B 0B   0B 0B 0B 0B 0B 0B 0B 0B  ..rJ............
recipeorganizer.net-startStop-1, WRITE: TLSv1 Handshake, length = 48
[Raw write]: length = 53
0000: 16 03 01 00 30 4A DD E4   1A B2 30 CF 7A 65 A4 3F  ....0J....0.ze.?
0010: 3F 90 1D 48 98 54 E4 E7   42 F6 1C 87 AA 70 3C 9E  ?..H.T..B....p<.
0020: F6 47 5E EC 40 D1 12 C3   EF 82 5E 44 09 F0 C9 A0  .G^.@.....^D....
0030: BB E4 E8 8C CF                                     .....
[Raw read]: length = 5
0000: 15 03 01 00 02                                     .....
[Raw read]: length = 2
0000: 02 14                                              ..
recipeorganizer.net-startStop-1, READ: TLSv1 Alert, length = 2
recipeorganizer.net-startStop-1, RECV TLSv1 ALERT:  fatal, bad_record_mac
%% Invalidated:  [Session-1, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA]

When the [Raw read]: length = 2 equals 2 it fails but if it equals 1 it succeeds. I am not a security expert at all so I'm not sure what might be causing this. Google and SO searches have not turned up anything useful. I did find one post that suggested I replace the local_policy and US_export_policy jars with the unlimited strength ones, but that did not fix the issue.

I believe I found the cause of the failure. This bug report appears to describe correctly the issue I was encountering: Inconsistent length for the generated secret using DH key agreement impl from SunJCE and PKCS11 . Every failure I examined had a PreMaster Secret key with 00 in the first byte, while every success had something else in the first byte (usually 01). The bug report indicates the fixed version is Java 8, so I'm assuming it was never fixed in Java 7, the version I'm using.

Until I'm able to upgrade to Java 8 I was able to force the JVM to not use any of the Diffie-Hellman cipher suites with a change to the java.security file:

jdk.tls.disabledAlgorithms=SSLv3, DH, DHE, ECDH, ECDHE

The file is located in the jre/lib/security subdirectory of the Java JDK directory on the server. An examination of the logs indicates one of the RSA cipher suites is now being used and the failure no longer occurs.

My only hesitation in declaring this the absolute cause is that one analysis of this bug indicated it should happen only 1 out of 256 handshakes, but I was seeing it occur almost 40-50% of the time. Regardless, after bypassing the DH cipher suites the exception no longer occurs.