Too many open file handles

Question

I'm working on a huge legacy Java application, with a lot of handwritten stuff, which nowadays you'd let a framework handle.

The problem I'm facing right now is that we are running out of file handles on our Solaris Server. I'd like to know what's the best way to track open file handles? Where to look at and what can cause open file handles to run out?

I cannot debug the application under Solaris, only on my Windows development environment. Is is even reasonable to analyze the open file handles under Windows?

Answer 1

It could certainly give you an idea. Since it's Java, the file open/close mechanics should be implemented similarly (unless one of the JVMs are implemented incorrectly). I would recommend using File Monitor on Windows.

Answer 2

Not a direct answer to your question but these problems could be the result of releasing file resources incorrectly in your legacy code. By example if you're working with FileOutputsStream classes make sure the close methods are called in a finally block as in this example:

FileOutputsStream out = null;
try {
  //You're file handling code
} catch (IOException e) {
  //Handle
} finally {
  if (out != null) {
    try { out.close(): } catch (IOException e) { }
  }
}

Answer 3

On windows you can look at open file handles using process explorer:

http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

On Solaris you can use "lsof" to monitor the open file handles

Answer 4

Its worth bearing in mind that open sockets also consume file handles on Unix systems. So it could very well be something like a database connection pool leak (e.g. open database connections not being closed and returned to the pool) that is leading to this issue - certainly I have seen this error before caused by a connection pool leak.

Answer 5

To answer the second part of the question:

what can cause open file handles to run out?

Opening a lot of files, obviously, and then not closing them.

The simplest scenario is that the references to whatever objects hold the native handles (e.g., FileInputStream) are thrown away before being closed, which means the files remain open until the objects are finalized.

The other option is that the objects are stored somewhere and not closed. A heap dump might be able to tell you what lingers where (jmap and jhat are included in the JDK, or you can use jvisualvm if you want a GUI). You're probably interested in looking for objects owning FileDescriptors.

Answer 6

This is a coding pattern that helps find unclosed resources. It closes the resources and also complains in the log about the problem.

class
{
    boolean closed = false;
    File file;

    close() {
        closed = true;
        file.close();
    }

    finalize() {
        if (!closed) {
            log error "OI! YOU FORGOT TO CLOSE A FILE!"
        file.close();
    }
}

Wrap the above file.close() calls in try-catch blocks that ignore errors.

Also, Java 7 has a new 'try-with-resource' feature that can auto-close resources.