Job keeps running in LocalJobRunner under Cloudera

Question

Need some quick help. Our job runs fine under MapR, but when we start the same job on Cloudera 5.1, it keeps running in Local mode.

I am sure this is some kind of configuration issue. Which config setting is it?

14/08/22 12:16:58 INFO mapreduce.Job: map 0% reduce 0%
14/08/22 12:17:03 INFO mapred.LocalJobRunner: map > map
14/08/22 12:17:06 INFO mapred.LocalJobRunner: map > map
14/08/22 12:17:09 INFO mapred.LocalJobRunner: map > map

Thanks.

Answer 1

Problem was that Cloudera 5.1 runs 'Yarn' distribution & under Yarn if you want to start a MapReduce job you SHOULD NOT use:

java -cp $CLASSPATH CLASSNAME (Under Yarn this will start job in the Local mode.)

Instead you've to use

yarn -cp $CLASSPATH CLASSNAME

Also, another thing I learned is that, you can get the HADOOP classpath by running the following command:

yarn classpath