Job keeps running in LocalJobRunner under Cloudera

问题:

Need some quick help. Our job runs fine under MapR, but when we start the same job on Cloudera 5.1, it keeps running in Local mode.

I am sure this is some kind of configuration issue. Which config setting is it?

14/08/22 12:16:58 INFO mapreduce.Job: map 0% reduce 0%
14/08/22 12:17:03 INFO mapred.LocalJobRunner: map > map
14/08/22 12:17:06 INFO mapred.LocalJobRunner: map > map
14/08/22 12:17:09 INFO mapred.LocalJobRunner: map > map

Thanks.

回答1:

Problem was that Cloudera 5.1 runs 'Yarn' distribution & under Yarn if you want to start a MapReduce job you SHOULD NOT use:

java -cp $CLASSPATH CLASSNAME (Under Yarn this will start job in the Local mode.)

Instead you've to use

yarn -cp $CLASSPATH CLASSNAME

Also, another thing I learned is that, you can get the HADOOP classpath by running the following command:

yarn classpath