Java multithreaded file downloading performance

问题:

Having recently worked on a project which required some more IO interaction than I'm used to, I felt like I wanted to look past the regular libraries (Commons IO, in particular) and tackle some more in depth IO issues.

As an academic test, I decided to implement a basic, multi-threaded HTTP downloader. The idea is simple: provide a URL to download, and the code will download the file. To increase download speeds, the file is chunked and each chunk is downloaded concurrently (using the HTTP Range: bytes=x-xheader) to use as much bandwidth as possible.

I have a working prototype, but as you may have guessed, it's not exactly ideal. At the moment I manually start 3 "downloader" threads which each download 1/3 of the file. These threads use a common, synchronized "file writer" instance to actually write the files to disk. When all threads are done, the "file writer" is completed and any open streams are closed. Some snippets of code to give you an idea:

The thread start-up:

ExecutorService downloadExecutor = Executors.newFixedThreadPool(3);
...
downloadExecutor.execute(new Downloader(fileWriter, download, start1, end1));
downloadExecutor.execute(new Downloader(fileWriter, download, start2, end2));
downloadExecutor.execute(new Downloader(fileWriter, download, start3, end3));

Each "downloader" thread downloads a chunk (buffered) and uses the "file writer" to write to disk:

int bytesRead = 0;
byte[] buffer = new byte[1024*1024];
InputStream inStream = entity.getContent();
long seekOffset = chunkStart;
while ((bytesRead = inStream.read(buffer)) != -1)
{
    fileWriter.write(buffer, bytesRead, seekOffset);
    seekOffset += bytesRead;
}

The "file writer" writes to disk using a RandomAccessFile to seek()and write() the chunks to disk:

public synchronized void write(byte[] bytes, int len, long start) throws IOException
{
      output.seek(start);
      output.write(bytes, 0, len);
}

All things considered, this approach seems to work. However, it doesn't work very well. I'd appreciate some advice/help/opinions on the following points. Much appreciated.

1. The CPU usage of this code is through the roof. It's using half my CPU (50% of each of the 2 cores) to do this, which is exponentially more than comparable downloading tools which barely stress the CPU at all. I'm a bit mystified as to where this CPU usage comes from, as I wasn't expecting this.
2. Usually, there seems to be 1 of the 3 threads that is lagging behind significantly. The other 2 threads will finish, after which it takes the third thread (which seems to be mostly the first thread with the first chunk) 30 or more seconds to complete. I can see from the task manager that the javaw process is still doing small IO writes, but I don't really know why this happens (I'm guessing race conditions?).
3. Despite the fact that I've chosen quite a big buffer (1MB), I get the feeling that the InputStream almost never actually fills the buffer, which causes more IO writes than I would like. I'm under the impression that in this scenario, it would be best to keep the IO access to a minimum, but I don't know for sure whether this is the best approach.
4. I realise Java may not be the ideal language to do something like this, but I'm convinced there's much more performance to be had than I get in my current implementation. Is NIO worth exploring in this case?

Note: I use Apache HTTPClient to do the HTTP interaction, which is where the entity.getContent() comes from (in case anyone is wondering).

回答1:

To answer my own questions:

1. The increased CPU usage was due to a while() {} loop that was waiting for the threads to finish. As it turns out, awaitTermination is a much better alternative to wait for an Executor to finish :)
2. (And 3 and 4) This seems to be the nature of the beast; in the end I achieved what I wanted to do by using careful synchronization of the different threads that each download a chunk of data (well, in particular the writes of these chunks back to disk).

回答2:

Presumably the Apache HTTP client will be doing some buffering, with a smaller buffer. It will need a buffer to read the HTTP header reasonably, and probably handling chunked encoding.

回答3:

My immediate thought for best performance on Windows would be to use IO completions ports. What I don't know is (a) whether there are similar concepts in other OSes, and (b) whether there are any suitable Java wrappers? If portability isn't important to you, though, it may be possible to roll your own wrapper with JNI.

回答4:

Set a very large socket receive buffer. But really your performance will be limited by the network bandwidth, not CPU bandwidth. All you're doing really is allocating 1/3 of the network bandwidth to each downloader. I'd be surprised if you get much benefit.