Java: TaskExecutor for Asynchronous Database Write

Question

I'm thinking of using Java's TaskExecutor to fire off asynchronous database writes. Understandably threads don't come for free, but assuming I'm using a fixed threadpool size of say 5-10, how is this a bad idea?

Our application reads from a very large file using a buffer and flushes this information to a database after performing some data manipulation. Using asynchronous writes seems ideal here so that we can continue working on the file. What am I missing? Why doesn't every application use asynchronous writes?

Answer 1

Why doesn't every application use asynchronous writes?

It's often necessary/usefull/easier to deal with a write failure in a synchronous manner.

Answer 2

I'm not sure a threadpool is even necessary. I would consider using a dedicated databaseWriter thread which does all writing and error handling for you. Something like:

 public class AsyncDatabaseWriter implements Runnable {
     private LinkedBlockingQueue<Data> queue = ....
     private volatile boolean terminate = false;

     public void run() {
         while(!terminate) {
            Data data = queue.take();
            // write to database
         }
     }
     public void ScheduleWrite(Data data) {
         queue.add(data);
     }
 }

I personally fancy the style of using a Proxy for threading out operations which might take a long time. I'm not saying this approach is better than using executors in any way, just adding it as an alternative.

Answer 3

Idea is not bad at all. Actually I just tried it yesterday because I needed to create a copy of online database which has 5 different categories with like 60000 items each.

By moving parse/save operation of each category into the parallel tasks and partitioning each category import into smaller batches run in parallel I reduced the total import time from several hours (estimated) to 26 minutes. Along the way I found good piece of code for splitting the collection: http://www.vogella.de/articles/JavaAlgorithmsPartitionCollection/article.html

I used ThreadPoolTaskExecutor to run tasks. Your tasks are just simple implementation of Callable interface.

Answer 4

why doesn't every application use asynchronous writes? - erm because every application does a different thing.

can you believe some applications don't even use a database OMG!!!!!!!!!

seriously though, given as you don't say what your failure strategies are - sounds like it could be reasonable. What happens if the write fails? or the db does away somehow

some databases - like sybase - have (or at least had) a thing where they really don't like multiple writers to a single table - all the writers ended up blocking each other - so maybe it wont actually make much difference...