Http java file download problem

Question

Trying to download file with apache httpclient library and have a problem with resulting file being smaller than the original (approximately 32-32kb, when normal file size is 92-93) and cannot be opened normally in pdf viewer. Can someone explain me why this can be happening ? (Using firefox to download this file can sometimes lead to file being downloaded fully and sometimes being downloaded partly)

Here is code I was using to download file via URL

    URL url = new URL("pathtofile");
    final URLConnection connection = url.openConnection();

    final InputStream is = connection.getInputStream();
    FileOutputStream fos = new FileOutputStream("C://result1.pdf");

    byte buffer[] = new byte[1024];
    int bytesRead; 
    while ((bytesRead = is.read(buffer)) >= 0) {        
        fos.write(buffer, 0, bytesRead);
    }
    fos.flush();
    fos.close();
    is.close();

P.S. Was trying to download this file using HttpClient apache library, same result.

UPDATED: Monitoring traffic with network tool I found the difference between receiving file via Firefox and application.

With Firefox first HttpPayloadLine was :

HTTPPayloadLine: 83 Td /F2 5.80476 Tf (A:\040Asinis\04017.12.10\04008:32\040laboratorij) Tj 100 Tz 1 1 1 rg /F1 5.80476 Tf 0 0 0 rg 104.4856 0 Td <0145> Tj 1 1 1 rg 0 0 0 rg 3.62799 0.72565 Td /F2 5.80476 Tf (\040) Tj 1 1 1 rg 0.83137 0.81569 0.78431 RG ET 51

With application first HttpPayload was

HTTPPayloadLine: CWgC,ú&ÿ3@Î"ݯV¨®~×>×)\ªleÚlµï½ci ¤Ãðð'È/CÈAø¯ª ÍübA«1Ãÿ Åç«VɬZòYóóy7»ÇH.o²e<qZna3l±°¥þ6ñþ[2YÚ1ì³Eë-ÓÊÏ$y:tÎà![ËÅS¤¿É¡¢è,þ|ºs¨)@¢Qâ¯ÝF~}oµÒ>¦ OAxz³äÒ.ß9 æÃZ¤ùÒ¨*«øUή+4×

This measurements was taken via Microsoft Network Monitor

LAST UPDATE It was a server problem after all, after they fixed that files are downloaded successful

Answer 1

The first thing that I spotted is that you check whether is.read(buffer) > 0, which is wrong, as it may (in theory, at least) return 0 even if it's not reached the end of file. InputStream.read() will return -1 when EOF is reached, so make that comparison >= 0.

EDIT: The second thing that I spotted (a little late, as it's already been noticed in other answers) is that you're writing the whole buffer to the output stream no matter how much of it was actually affected by the latest read operation. Try something like:

byte[] buffer = new byte[BUFFER_SIZE];
int bytesRead;
while ( (bytesRead = in.read(buffer)) >= 0 ) {
    out.write(buffer, 0, bytesRead);
}

Answer 2

Try changing to

while ((bytesRead = in.read(buffer)) != -1) {
 byte[] tmp = ArrayUtils.subarray(buffer, 0, bytesRead);
 fos.write(tmp);
}

you mite get 0 bytes back but that does not mean its finished.Also write only bytes that you received not buffer.

Answer 3

Maybe reading the error stream can give you some information:

connection.getErrorStream();

Answer 4

Can you use org.apache.commons.io.FileUtils.copyURLToFile(URL, File) instead?