What is the difference between classic, local for

问题:

The official description for this parameter is as follows:

The runtime framework for executing MapReduce jobs. Can be one of local, classic or yarn.

I know that the value 'yarn' is for MRv2, which will submit the mapreduce job to resourcemanager. But what is the difference between local and classic? Which one is corresponding to MRv1?

Thanks a lot!

回答1:

You are right, "yarn" stands for MRv2. "classic" is for MRv1 and "local" for local runs of the MR jobs. But why do you need MRv1? Yarn is out of the beta now and it is more stable than old MRv1 framework, while your MapReduce jobs still can use old "mapred" API

回答2:

I agree with the above answer, I would like to add one more point.

Classic is MR1, whenever we submit MR jobs using framework name as classic, Job will be submitted to Jobtracker daemon in MR1 which will coordinate the mapreduce execution, each tasks will be executed in different JVMs

Whereas the main purpose of local JobRunner is debugging/testing mapreduce program with small inputs. It doesn't need any daemons like JobTracker, TaskTracker. This execution mode is useful when you execute any MR application from eclipse, by default execution will be in local jobrunner(Uses the same JVM that eclipse use). All mapper/reduce will be executing the same JVM. As the same JVM is being used for all executions(Driver+Map+Reduce), one cannot use this for processing large data, execution will end up with OutOfMemory exception.