Resource leak in Files.list(Path dir) when stream

Question

I recently wrote a small app that periodically checked the content of a directory. After a while, the app crashed because of too many open file handles. After some debugging, I found the error in the following line:

Files.list(Paths.get(destination)).forEach(path -> {
     // To stuff
});

I then checked the javadoc (I probably should have done that earlier) for Files.list and found:

* <p> The returned stream encapsulates a {@link DirectoryStream}.
* If timely disposal of file system resources is required, the
* {@code try}-with-resources construct should be used to ensure that the
* stream's {@link Stream#close close} method is invoked after the stream
* operations are completed

To me, "timely disposal" still sounds like the resources are going to be released eventually, before the app quits. I looked through the JDK (1.8.60) code but I wasn't able to find any hint about the file handles opened by Files.list being released again.

I then created a small app that explicitly calls the garbage collector after using Files.list like this:

while (true) {
    Files.list(Paths.get("/")).forEach(path -> {
      System.out.println(path);
    });
    Thread.sleep(5000);

    System.gc();
    System.runFinalization();
}

When I checked the open file handles with lsof -p <pid> I could still see the list of open file handles for "/" getting longer and longer.

My question now is: Is there any hidden mechanism that should eventually close no longer used open file handles in this scenario? Or are these resources in fact never disposed and the javadoc is a bit euphemistic when talking about "timely disposal of file system resources"?

Answer 1

Regarding the IDE part: Eclipse performs resource leak analysis based on local variables (and explicit resource allocation expressions), so you only have to extract the stream to a local variable:

Stream<Path> files =Files.list(Paths.get(destination));
files.forEach(path -> {
 // To stuff
});

Then Eclipse will tell you

Resource leak: 'files' is never closed

Behind the scenes the analysis works with a cascade of exceptions:

1. All Closeables need closing
2. java.util.stream.Stream (which is Closeable) does not need closing
3. All streams produced by methods in java.nio.file.Files do need closing

This strategy was developed in coordination with the library team when they discussed whether or not Stream should be AutoCloseable.

Answer 2

If you close the Stream, Files.list() does close the underlying DirectoryStream it uses to stream the files, so there should be no resource leak as long as you close the Stream.

You can see where the DirectoryStream is closed in the source code for Files.list() here:

return StreamSupport.stream(Spliterators.spliteratorUnknownSize(it, Spliterator.DISTINCT), false)
                    .onClose(asUncheckedRunnable(ds));

The key thing to understand is that a Runnable is registered with the Stream using Stream::onClose that is called when the stream itself is closed. That Runnable is created by a factory method, asUncheckedRunnable that creates a Runnable that closes the resource passed into it, translating any IOException thrown during the close() to an UncheckedIOException

You can safely assure that the DirectoryStream is closed by ensuring the Stream is closed like this:

try (Stream<Path> files = Files.list(Paths.get(destination))){
    files.forEach(path -> {
         // Do stuff
    });
}

Answer 3

   List<String> fileList = null;

   try (Stream<Path> list = Files.list(Paths.get(path.toString()))) {
   fileList = 
     list.filter(Files::isRegularFile).map(Path::toFile).map(File::getAbsolutePath)
                                .collect(Collectors.toList());
   } catch (IOException e) {
    logger.error("Error occurred while reading email files: ", e);
  }