excessive memory use by java

Question

In a project of mine I constantly compress little blocks of data. Now I find out that the jvm then grows to 6GB of ram (resident (RES) RAM, not shared or virtual or so) and then die because of out of memory. It is as if the garbage collector never runs or so. I've pulled out the relevant code and pasted it below. When I run it (java6, 32 bit linux) it grows to 1GB of ram. Anyone got an idea how to reduce the memory usage?

import java.util.Random;
import java.util.zip.Deflater;
import java.util.zip.Inflater;

class test  {
    int blockSize = 4096;
    Random r = new Random();

    public test() throws Exception {
        blockSize = 4096;
        byte [] data = new byte[blockSize];
        for(int index=0; index<blockSize; index++)
            data[index] = (byte)r.nextInt();

        for(long cnt=0; cnt<1000000; cnt++) {
            byte [] result = compress(data);
            if (result != null)
                data[0] = result[0];
        }
    }

    byte [] compress(byte [] in) {
        assert in.length == blockSize;

        Deflater compresser = new Deflater();
        compresser.setInput(in);
        compresser.finish();
        byte [] out = new byte[in.length];
        int outLen = compresser.deflate(out);

        if (outLen < blockSize) {
            byte [] finalOut = new byte[outLen];
            System.arraycopy(out, 0, finalOut, 0, outLen);
            return finalOut;
        }

        return null;
    }

    public static void main(String [] args) throws Exception {
        new test();
    }
}

Answer 1

Well, It seems to me that there is no memory leak in the code, so it actually seems the VM is not GC-ing byte arrays.

"Anyone got an idea how to reduce the memory usage?"
Well, I would try with

byte firstByteOfDataWhichIsCompressedAndThenUncompressed(byte [] in) { ... }

which specifically returns the first byte of the uncompressed array, rather than the whole array. I know, it's a horrible method name, and I hope you will find a better one.

The following code

    for(long cnt=0; cnt<1000000; cnt++) {
        byte [] result = compress(data);
        if (result != null)
            data[0] = result[0];
    }

would become

    for(long cnt=0; cnt<1000000; cnt++)
        data[0] = firstByteOfDataWhichIsCompressedAndThenUncompressed(data);

Answer 2

Well, Folkert van Heusden solved his own problem, but to summarize:

Early in the compress(byte [] in)-method, we create a java.util.zip.Deflater.

We use the Deflater to do some stuff, and then we leave the compress()-method. We loose our reference to the deflater-variable. At this point, the Deflater is no longer in use, and is waiting to be killed by the garbage collector.

Deflater allocates both Java heap memory and C/C++/native heap memory. The native heap memory that are allocated by a Deflater, will be held until Deflater.finalize-method is called by the garbage collector. If the garbage collector doesn't run fast enough (there might be plenty free java heap memory), we can run out of C/C++ heap memory. If this happen, we will get "Out of memory"-errors.

The Oracle bug report JDK-4797189 is probably related. It contains a code snippet that illustrates and reproduces the problem:

public class Bug {
    public static void main( String args[] ) {
        while ( true ) {
            /* If ANY of these two lines is not commented, the JVM
             runs out of memory */
            final Deflater deflater = new Deflater( 9, true );
            final Inflater inflater = new Inflater( true );
        }
    }
}

The solution is to free the resources when you are finished by calling the Deflater.end()-method (or Inflater.end()).