I am trying to debug a few slow responses served by an app deployed on Tomcat.
Right now I am focussing on SecureRandom
and /dev/random
(some of the other probable causes have been investigated and ruled out).
The pattern is as follows:
- The first call takes exactly 30.0xy seconds after Tomcat restart (even if the request arrives 4 minutes after the Startup)
- Later, some calls take exactly 15.0pq seconds (there was no specific pattern that I could establish, pq being the time approximate time taken in TP99)
The service call involves encryption and decryption (AES/ECB/PKCS5Padding).
Is it possible that SecureRandom init/repopulating is leading to this?
(Although, there is a log written in catalina.log that says "Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [28,760] milliseconds."
)
Also, in order to check whether /dev/random
or /dev/urandom
is being used, I used the test from this question. To my surprise, I didn't see reads from either of them unlike the way it happens in the linked question.
These are the last few lines from the strace
log:
3561 lstat("/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/jsse.jar", {st_mode=S_IFREG|0644, st_size=258525, ...}) = 0
3561 open("/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/jsse.jar", O_RDONLY) = 6
3561 stat("/dev/random", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 8), ...}) = 0
3561 stat("/dev/urandom", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 9), ...}) = 0
3561 open("/dev/random", O_RDONLY) = 7
3561 open("/dev/urandom", O_RDONLY) = 8
3561 unlink("/tmp/hsperfdata_xxxx/3560") = 0
What is then being used for seeding SecureRandom?
fyi, java -version
java version "1.6.0_32"
OpenJDK Runtime Environment (IcedTea6 1.13.4) (rhel-7.1.13.4.el6_5-x86_64)
OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)
I could not check your OpenJDK concrete version, but I could check jdk6-b33.
SecureRandom uses SeedGenerator to get the seed bytes
public byte[] engineGenerateSeed(int numBytes) {
byte[] b = new byte[numBytes];
SeedGenerator.generateSeed(b);
return b;
}
SeedGenerator gets the seedSource
(String) from SunEntries
String egdSource = SunEntries.getSeedSource();
SunEntries
tries to get the source from the system property java.security.egd
first, if is not found then tries to get the property securerandom.source
from the java.security
properties file, if the property is not found returns a blank string.
// name of the *System* property, takes precedence over PROP_RNDSOURCE
private final static String PROP_EGD = "java.security.egd";
// name of the *Security* property
private final static String PROP_RNDSOURCE = "securerandom.source";
final static String URL_DEV_RANDOM = "file:/dev/random";
final static String URL_DEV_URANDOM = "file:/dev/urandom";
private static final String seedSource;
static {
seedSource = AccessController.doPrivileged(
new PrivilegedAction<String>() {
public String run() {
String egdSource = System.getProperty(PROP_EGD, "");
if (egdSource.length() != 0) {
return egdSource;
}
egdSource = Security.getProperty(PROP_RNDSOURCE);
if (egdSource == null) {
return "";
}
return egdSource;
}
});
}
the SeedGenerator
check this value to initialize the instance
// Static instance is created at link time
private static SeedGenerator instance;
private static final Debug debug = Debug.getInstance("provider");
final static String URL_DEV_RANDOM = SunEntries.URL_DEV_RANDOM;
final static String URL_DEV_URANDOM = SunEntries.URL_DEV_URANDOM;
// Static initializer to hook in selected or best performing generator
static {
String egdSource = SunEntries.getSeedSource();
// Try the URL specifying the source
// e.g. file:/dev/random
//
// The URL file:/dev/random or file:/dev/urandom is used to indicate
// the SeedGenerator using OS support, if available.
// On Windows, the causes MS CryptoAPI to be used.
// On Solaris and Linux, this is the identical to using
// URLSeedGenerator to read from /dev/random
if (egdSource.equals(URL_DEV_RANDOM) || egdSource.equals(URL_DEV_URANDOM)) {
try {
instance = new NativeSeedGenerator();
if (debug != null) {
debug.println("Using operating system seed generator");
}
} catch (IOException e) {
if (debug != null) {
debug.println("Failed to use operating system seed "
+ "generator: " + e.toString());
}
}
} else if (egdSource.length() != 0) {
try {
instance = new URLSeedGenerator(egdSource);
if (debug != null) {
debug.println("Using URL seed generator reading from "
+ egdSource);
}
} catch (IOException e) {
if (debug != null)
debug.println("Failed to create seed generator with "
+ egdSource + ": " + e.toString());
}
}
// Fall back to ThreadedSeedGenerator
if (instance == null) {
if (debug != null) {
debug.println("Using default threaded seed generator");
}
instance = new ThreadedSeedGenerator();
}
}
if the source is
final static String URL_DEV_RANDOM = "file:/dev/random";
or
final static String URL_DEV_URANDOM = "file:/dev/urandom"
uses the NativeSeedGenerator
, on Windows tries to use the native CryptoAPI
on Linux the class simply extends the SeedGenerator.URLSeedGenerator
package sun.security.provider;
import java.io.IOException;
/**
* Native seed generator for Unix systems. Inherit everything from
* URLSeedGenerator.
*
*/
class NativeSeedGenerator extends SeedGenerator.URLSeedGenerator {
NativeSeedGenerator() throws IOException {
super();
}
}
and call to the superclass constructor who loads /dev/random
by default
URLSeedGenerator() throws IOException {
this(SeedGenerator.URL_DEV_RANDOM);
}
so, OpenJDK uses /dev/random
by default until you do not set another value in the system property java.security.egd
or in the property securerandom.source
of security properties file.
If you want to see the read results using strace
you can change the command line and add the trace=open,read
expression
sudo strace -o a.strace -f -e trace=open,read java class
the you can see something like this (I did the test with Oracle JDK 6)
13225 open("/dev/random", O_RDONLY) = 8
13225 read(8, "@", 1) = 1
13225 read(3, "PK\3\4\n\0\0\0\0\0RyzB\36\320\267\325u\4\0\0u\4\0\0 \0\0\0", 30) = 30
....
....
The Tomcat Wiki section for faster startup suggest using a non-blocking entropy source like /dev/urandom if you are experiencing delays during startup
More info: https://wiki.apache.org/tomcat/HowTo/FasterStartUp#Entropy_Source
Hope this helps.
The problem is not SecureRandom per se but that /dev/random blocks if it doesn't have enough data. You can use urandom instead but that might not be a good idea if you need cryptographically strong random seeds.
On headless Linux systems you can install the haveged daemon. This keeps /dev/random topped up with enough data so that calls don't have to wait for the required entropy to be generated.
I've done this on a Debian Aws instance and watched SecureRandom generateBytes calls drop from 25 seconds to sub millisecond (Openjdk 1.7 something, can't remember specifically what version).