How to improve a Java application working with Bas

Question

We have an issue in our server at job and I'm trying to understand what is happening. It's a Java application that runs in a linux server, the application recieve inforamtion form TCP socket and analyse them and after analyse write into the database.

Sometimes the quantity of packets is too many and the Java application need to write many times into the database per second (like 100 to 500 times).

I try to reproduce the issue in my own computer and look how the application works with JProfiler.

The memory look always going up, is it a memory leak (sorry I'm not a Java programmer, i'm C++ programmer)?

After 133 minute

After 158 minute

I have many locked thread, does it means that the application did not programmed correctly?

Is it too many connection to the database (the application use BasicDataSource class to use a connection pool)?

The program don't have FIFO to manage database writing for continual information entering from TCP port. My questions are (remeber that I'm not a Java programmer and I don't know if this is way that a Java application should work or the program can be programmed more efficient)

1. Do you think that something is wrong with the code that are not correctly managing write, read, updates on the database and cosume too many memory and CPU time, or is it the way that it works in BasicDataSource class?

2. How do you think I can improve it (if you think it's an issue) this issue, by creating a FIFO and removing the part of code that create too many threads? Or the threads is not the application threads himself and thats the BasicDataSource threads?

Answer 1

There are several areas to dig into, but first I would try and find what is actually blocking the threads in question. I'll assume everything before the app is being looked at as well, so this is from the app down.

I know the graphs show free memory but they are just point in time so I can't see a trend. GC logging is available, I haven't used JProfiler much though so I am not sure how to point you to it in that tool. I know in DynaTrace I can see GC events and their duration as well as any other blocking events and their root cause as well. If this isn't available there are command line switches to log GC activity to see its duration and frequency. That is one area that could block.

I would also look at how many connections you have in your pool. If there are 100-500 requests/second trying to write and they are stacking up because you don't have enough connections to work them then that could be a problem as well. The image shows all transactions but doesn't speak to the pool size. Transactions blocked with nowhere to go could lead to your memory jumps as well.

There is also the flip side that your database can't handle the traffic and is pegged, and that is what is blocking the connections as well so you would want to monitor that end of things and see if that is a possible cause of the blocking.

There is also the chance that the blocking is occurring from the SQL being run as well, waiting for page locks to be released, etc.

Lots of areas to look at, but I would address and verify one layer at a time starting with the app and working down.