Performance drop after 5 days running web applicat

Question

I've developed a web application using the following tech stack:

• Java
• Mysql
• Scala
• Play Framework
• DavMail integration (for calender and exchange server)
• Javamail
• Akka actors

On the first days, the application runs smoothly and without lags. But after 5 days or so, the application gets really slow! And now I have no clue how to profile this, since I have huge dependencies and it's hard to reproduce this kind of thing. I have looked into the memory and it seems that everything its okay.

Any pointers on the matter?

Answer 1

Yet another profiler is Yourkit.
It is commercial, but with trial period (two weeks). Actually, I've firstly tried VisualVM as @axel22 suggested, but our remote server was ssh'ed and we had problems with connecting via VisualVM (not saying that it is impossible, I've just surrendered after a few hours).

Answer 2

jconsole comes with the JDK and is an easy tool to spot bottlenecks. Connect it to your server, look into memory usage, GC times, take a look at how many threads are alive because it could be that the server creates many threads and they never exit.

Answer 3

You might just want to try the 'play status' command, which will list web app state (threads, jobs, etc). This might give you a hint on what's going on.

Answer 4

So guys, in this specific case, I was running play in Developer mode, which makes the compiler works every now and then.

After changing to production mode, everything was lightning fast and no more problems anymore. But thanks for all the help.

Answer 5

I agree with tulskiy. On top of that you could also use JMeter if the investigations you will have made with jconsole are unconclusive.

The probable causes of the performances degradation are threads (that are created but never exit) and also memory leaks: if you allocate more and more memory, before having the OutOfMemoryError, you may encounter some performances degradation (happened to me a few weeks ago).

Answer 6

`visualvm˙ is also a great tool for such purposes, you can connect to a remote JVM as well and see what's inside.

I suggest you doing this:

• take a snapshot of the application running since few hours and since 5 days
• compare thread counts
• compare object counts, search for increasing numbers
• see if your program spends more time in particular methods on the 5th day than on the 1str one
• check for disk space, maybe you are running out of it