Overriding default hadoop jars in class path

问题:

I've seen many manifestations of ways to use the user class path as precedent to the hadoop one. Often times this is done if an m/r job needs a specific version of a library that hadoop coincidentally already uses an older version of (for example jackson's json parser or commons http , etc.)

In any case : I've seen :

mapreduce.task.classpath.user.precedence
mapreduce.task.classpath.first
mapreduce.job.user.classpath.first

Which one of these parameters is the right one to set in my job configuration, in order to force mappers and reducers to have a class path which puts my user defined hadoop_classpath jars BEFORE the hadoop default dependency jars ?

By the way, this is related to this question : Dynamodb requestHandler acception which I recently have found is due to a jar conflict.

回答1:

So, assuming you're using 0.20.203, this is handled in the TaskRunner.java code as follows:

• The property you're looking for is on line 94 - mapreduce.user.classpath.first
• Line 214 is where the call is made to build the list of classpaths, which delegates to a method called getClassPaths(..)
• getClassPaths() is defined on line 524, and you should be able to see that the configuration property is used to decide on whether your job + dist cache libraries, or the hadoop libraries go on the classpath first

For other versions of hadoop, you're best to check the TaskRunner.java class to confirm the name of the config property after all this is a "semi hidden config":

static final String MAPREDUCE_USER_CLASSPATH_FIRST =
        "mapreduce.user.classpath.first"; //a semi-hidden config

回答2:

As in the latest Hadoop version (2.2+), you should set:

    conf.setBoolean(MRJobConfig.MAPREDUCE_JOB_USER_CLASSPATH_FIRST, true);

回答3:

These settings work for referencing classes of external jars only in your mapper or reducer tasks. If, however, you are using these in, for example a customized InputFormat, it will fail to load the class. A way to make sure this also works everywhere (in MR2) is exporting this setting when submitting your job:

export HADOOP_USER_CLASSPATH_FIRST=true

回答4:

I had the same issue and the parameter that worked for me on Hadoop Version 0.20.2-cdhu03 is "mapreduce.task.classpath.user.precedence"

This setting is tested not work on CDH3U3, following answer is from Cloudera team:

// JobConf job = new JobConf(getConf(), MyJob.class);
// job.setUserClassesTakesPrecedence(true);

http://archive.cloudera.com/cdh/3/hadoop/api/org/apache/hadoop/mapred/JobConf.html#setUserClassesTakesPrecedence%28boolean%29

回答5:

In the MapR distribution, the property is "mapreduce.task.classpath.user.precedence"
http://www.mapr.com/doc/display/MapR/mapred-site.xml

<property>
    <name>mapreduce.task.classpath.user.precedence</name>
    <value>true</value>
    <description>Set to true if user wants to set different classpath. (AVRO) </description>
</property>

jobConf.setUserClassesTakesPrecedence(true);