java: read large binary file

Question

I need to read out a given large file that contains 500000001 binaries. Afterwards I have to translate them into ASCII.

My Problem occurs while trying to store the binaries in a large array. I get the warning at the definition of the array ioBuf:

"The literal 16000000032 of type int is out of range."

I have no clue how to save these numbers to work with them! Has somebody an idea?

Here is my code:

public byte[] read(){
    try{
        BufferedInputStream in = new BufferedInputStream(new FileInputStream("data.dat"));
        ByteArrayOutputStream bs = new ByteArrayOutputStream();
        BufferedOutputStream out = new BufferedOutputStream(bs);
        byte[] ioBuf = new byte[16000000032];       
        int bytesRead;
        while ((bytesRead = in.read(ioBuf)) != -1){
            out.write(ioBuf, 0, bytesRead);
        }
          out.close();
          in.close();
          return bs.toByteArray();
}

Answer 1

1. If you need to declare a large constant, append an 'L' to it which indicates to the compiler that is a long constant. However, as mentioned in another answer you can't declare arrays that large.
2. I suspect the purpose of the exercise is to learn how to use the java.nio.Buffer family of classes.

Answer 2

I made some progress by starting from scratch! But I still have a problem.

My idea is to read up the first 32 bytes, convert them to a int number. Then the next 32 bytes etc. Unfortunately I just get the first and don't know how to proceed.

I discovered following method for converting these numbers to int:

public static int byteArrayToInt(byte[] b){
    final ByteBuffer bb = ByteBuffer.wrap(b);
    bb.order(ByteOrder.LITTLE_ENDIAN);
    return bb.getInt();
}

so now I have:

    BufferedInputStream in=null;
    byte[] buf = new byte[32];
    try {
        in = new BufferedInputStream(new FileInputStream("ndata.dat"));
        in.read(buf);
        System.out.println(byteArrayToInt(buf));
        in.close();
    } catch (IOException e) {
        System.out.println("error while reading ndata.dat file");
    }

Answer 3

The maximum Index of an Array is Integer.MAX_VALUE and 16000000032 is greater than Integer.MAX_VALUE

Integer.MAX_VALUE = 2^31-1 = 2147483647

2147483647 < 16000000032

You could overcome this by checking if the Array is full and create another and continue reading. But i'm not quite sure if your approach is the best way to perform this. byte[Integer_MAX_VALUE] is huge ;) Maybe you can split the input file in smaller chunks process them.

EDIT: This is how you could read a single int of your file. You can resize the buffer's size to the amount of data you want to read. But you tried to read the whole file at once.

//Allocate buffer with 4byte = 32bit = Integer.SIZE
byte[] ioBuf = new byte[4];       
int bytesRead;
while ((bytesRead = in.read(ioBuf)) != -1){
   //if bytesRead == 4 you read 1 int
   //do your stuff
}