Why does logstash take so long to start/load?

Question

Edit : I changed the title because the issue was not what I initially thought. The fact is that logstash takes more than a minute to starts, which can be misinterpreted as "silence"...

---

I'm trying to make logstash running, so I've followed the instruction on the official site for a standalone installation : http://logstash.net/docs/1.1.4/tutorials/getting-started-simple

Basically, I got the logstash-1.1.4-monolithic.jar file, then build a very simple config file : (example.conf)

input {
  stdin { type => "stdin-type"  }
}
output {
  stdout { debug_format => "json" }
}

But when I run logstash, nothing is coming out (I'm typing random text in STDIN, but get no response) :

# java -jar logstash-1.1.4-monolithic.jar agent -f example.conf
Test
toto
hey ??? Wakeup !!!
^C

(For information : Java version is correct)

# java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) Client VM (build 20.1-b02, mixed mode, sharing)

Please, can someone tell me what I'm missing ?

Answer 1

I started using logstash about a week ago. Was really seduced by @jordansissel hate-driven-development talks. Spent last days on the #logstash irc talking to ppl regarding issues including the one you have mentioned.

Starting logstash-web (a sparse sinatra web server) takes 29(!) seconds on my ubuntu 12.04.

I believe logstash has to move away from using java. Logstash can surely be running as ruby app like sensu and deliver same value with less (or no) pain.

With java(jruby) puts "bla" on console takes 7 seconds.

The short answer is: java is good in the cup, not in the computer. If you see something like "Prerequisite: Java" - prepare for waste.

Answer 2

Ok, I've found by myself.

Everything was working just fine. It's just that logstash is soooooo long to launch. More than 60 seconds on my (humble) server !! Add to that huge starting time the fact that nothing is printed when launched...

Answer 3

Because this question is still relevant, I should like to point out that if you're running on Java <8, then one reason you might get very slow start times on Linux (and perhaps Solaris as well) is because the Oracle JVM on those platforms has a bug with regard to where it is gettings its random numbers.

Even if told to get its random numbers from /dev/urandom, it won't (at least, not entirely?). If you get it to read from /dev/./urandom, which is the same, but doesn't match the internal broken logic, then you may well find that Java starts much faster (particularly in J2EE middleware environments).

http://bugs.java.com/view_bug.do?bug_id=6521844

That said, even with Java 8, logstash --configtest is still super slow to launch; this is likely due to the fact that a 64-bit JVM in server mode will do a lot of work on optimizing at startup (even then, I suspect JRuby is doing a lot of Ruby-related things).

The JRuby wiki has a document about other startup performance improvements (https://github.com/jruby/jruby/wiki/Improving-startup-time), but not very applicable to getting logstash much faster. I could get startup time for 'logstash --configtest' down to about 8 seconds with this:

[root@node-2 ~]# DEBUG=1 /opt/logstash/bin/logstash help
DEBUG: exec /opt/logstash/vendor/jruby/bin/jruby --1.9 -J-XX:+UseParNewGC -J-XX:+UseConcMarkSweepGC -J-Djava.awt.headless=true -J-XX:CMSInitiatingOccupancyFraction=75 -J-XX:+UseCMSInitiatingOccupancyOnly -J-XX:+HeapDumpOnOutOfMemoryError -J-Xmx1g -J-XX:HeapDumpPath=/opt/logstash/heapdump.hprof /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb help

Take the command it will exec, remove all the java options, and replace with --dev

[root@node-2 ~]# time /opt/logstash/vendor/jruby/bin/jruby /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent --configtest --config /etc/logstash/logstash-submission.conf 
Configuration OK

real    0m13.367s
user    0m12.966s
sys 0m0.321s
[root@node-2 ~]# time /opt/logstash/vendor/jruby/bin/jruby --dev /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent --configtest --config /etc/logstash/logstash-submission.conf 
Configuration OK

real    0m6.954s
user    0m6.629s
sys 0m0.286s

7 seconds still ain't great, but its better than 14 or so. 7 seconds is around the sweet spot for a sip of coffee, so avoids the awkward middle of your test finishing before your sip is complete. B^)

A simple alias may be a useful quick win:

alias logstash-configtest="/opt/logstash/vendor/jruby/bin/jruby --dev /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent --configtest"

I believe you can also run logstash with the USE_RUBY=1 flag to use the 'ruby' instead of the vendored jruby -- you wouldn't do this for running logstash normally, but it might be useful for --configtest if you want to use it frequently. -- you do need to make sure you have all the various Ruby modules, and then you'd have a version skew between your native ruby and the vendored jruby... But perhaps a Ruby buff could take this idea and run with it.

Answer 4

I think you jump to conclusions, just activate --debug logs and see whats going on + you can even use your kit to profile it and see exactly what goes own.