Parallel programming in Java

Question

How can we do Parallel Programming in Java? Is there any special framework for that? How can we make the stuff work?

I will tell you guys what I need, think that I developed a web crawler and it crawls a lot of data from the internet. One crawling system will not make things work properly, so I need more systems working in parallel. If this is the case can I apply parallel computing? Can you guys give me an example?

Answer 1

In java parallel processing is done using threads which are part of the runtime library

The Concurrency Tutorial should answer a lot of questions on this topic if you're new to java and parallel programming.

Answer 2

You can have a look at Hadoop and Hadoop Wiki.This is an apache framework inspired by google's map-reduce.It enables you to do distributed computing using multiple systems.Many companies like Yahoo,Twitter use it(Sites Powered By Hadoop).Check this book for more information on how to use it Hadoop Book.

Answer 3

This is the parallel programming resource I've been pointed to in the past:

http://www.jppf.org/

I have no idea whether its any good or not, just that someone recommended it a while ago.

Answer 4

I have heard about one at conference a few years ago - ParJava. But I'm not sure about the current status of the project.

Answer 5

Parallelism

Parallelism means that an application splits its tasks up into smaller subtasks which can be processed in parallel, for instance on multiple CPUs at the exact same time.

Answer 6

You might want to check out Hadoop. It's designed to have jobs running over an arbitrary amount of boxes and takes care of all the bookkeeping for you. It's inspired by Google's MapReduce and their related tools and so it even comes from web indexing.